'add error handling for categories.pm'

Detect and raise a fatal error if the script fails

Print the categories

'add get-list-of-current-categories.sh back in'

'-m'

Commit a work in progress, only to be deleted on my next commit

Remove 'svn commit checking status' since we don't do subversion commits any
more

Document that FreshPorts::Config::FreshPortsURL must have a trailing /

Use FreshPorts again...

Use FreshPorts::

'fix Makefile.PL'

'Create FreshPortsVerify'

'Create FreshPortsVerify'

Move

Move from FreshPorts::Scripts -> FreshPorts::Modules

'Create 1irAdd Makefile.pl'

'Create first tag'

'bringing in better category detection'

Stop using IgnoredItems and use a list of Categories pulled from the repo

Add categories.pm used thusly:

use FreshPorts::categories
use List::MoreUtils 'any';

FreshPorts::categories::FetchAll();

# look for special files outside a port, such as LEGAL, GIDs, UIDs, .hooks
if ( ! any {/$category_name/} @FreshPorts::Categories::categories ) {
...
}

With IgnoredItems being removed, this script is no longer required.

FreshPorts::Config::scriptpath and
FreshPorts::Config::ScriptDir have the same value. use only the latter.

Fix the debug output so it makes sense when reading it.

'add get-list-of-current-categories.sh; fix vuln_latest.pl for new HTML usage'

Commit my changes to vuln_latest.pl before incorporating changes from
https://github.com/FreshPorts/freshports/pull/261

'if no pkg-message in the repo, try apply-slist - corrected'

space changes only

add make-apply-slist.sh

'Previous release was the wrong tarball name - updated config.sh.sample'

A new iteration of the find

re https://github.com/FreshPorts/freshports/issues/245

Not all ports with a pkg-message have a pkg-message file.

Some have files/pkg-message.in, for example.

This script uses apply-slist to build that file.

'count *.xml not *.txt'

Now that the FreeBSD git repo is out, it seems they changed the name of the
head branch from master to main. The main change there is:

-my %BranchConversions = ( 'master' => 'head' );
+my %BranchConversions = ( 'main' => 'head' );

Adjust some constants to allow commits to process.

$FreshPorts::Config::RepoDir . '/PORTS-head'; was renamed to 
$FreshPorts::Config::RepoDir . '/' . $FreshPorts::Config::Repo_PORTS;

This meant that $FreshPorts::Config::Repo_PORTS had to be moved higher
up withint config.pm

Adjust the constants for the repo directories. We have abandoned github
and moved to git.freebsd.org.  e.g. freebsd-doc becomes doc

add DBUSER_REPORTING

'Add OPTIONS_NAME to scripts.
fix ports_tree_file_count.pl to work with new configuration items'

FreshPorts::Constants::Repo_Ports is gone.

Use FreshPorts::Constants::Repo_Dir_Name_Ports instead

Start saving options_name

Add new committer and author fields:

* committer_name
* committer_email
* author_name
* author_email

Add p5-Email-Address-XS>0:mail/p5-Email-Address-XS as new dependecy because
we need to split the user off the address.

We now take in Author and Committer.

Author is not yet written to the database.

'bring in the vuxml code from trunk'

Remove use of database

Sort the keys before displaying them so every host has the same order of
output.

In /var/db/ingress/message-queues/incoming, we look for *.xml

Remove use of database

use -mindepth 1 to avoid counting the directory itself

'K  IW   svn-commit.tmp (Modified)                                                                                                                                                                                                                      Row 3    Col 1   
Stop using FreshPorts::Utilities::CommitCountPeriod because we are counting
only file more than 5 minutes old.'

Stop using FreshPorts::Utilities::CommitCountPeriod because we are counting
only file more than 5 minutes old.

adjust report_non_zero to cover all incoming and spooling directories. same for CountRecent

trailing / issues, add in src-stable-12

Check for and error out if config.sh is not found

'fix system_status.sh'

'email files are *.txt not, not *.txt.raw'

'Show reasons for update and number of updates'

'allow IO::String on incoming new()'

Add three new paramaters:

* dryrun        - do not update the database
* showchecksums - show the checksums
* printnodes    - print the node we are processing

Change from IO::File to IO::String for the $nodeString = $node->toString();
XML. This means the fh->open($nodeString) succeeds.

'For new security/vuxml/vuln.xml layout'

'For new security/vuxml/vuln.xml layout'

Adjust to XML processing, not XML creation.
XML will be created by another user.

Change to fixed length wait times.
We are going to very high values for Fetch_Retry_Limit because svn is lagged
on replication. It is easier to wait than to error out and fix by hand.

Move to using XML::DOM::XPath now that security/vuxml/vuln.xml is split into
several files. Proof of concept courtesy of bapt@FreeBSD.org

Saving before deleting.  Work will be done by vuxml_parsing.pm

'update to 2.0.1'

add a comment showing an example make-port.sh call

add in stuff for the quarterly ports, and change tabs to spaces

'first cut of the scripts for git'

adjust spaces to tabs

Remove files no longer used.

Convert from `stuff` to $(stuff)

slightly different comments

Convert from `stuff` to $(stuff)

Revert last commit. It seems we need some of that code afterall.

Now that I've realized the website should process both svn and git commits,
I wanted a way to inspect an XML file and determine which type of repo it
came from. This is vital because we have two different sets of code to deal
with each origin:

[dan@devgit-ingress01:~/scripts] $ ls -l load_xml_into_db*
-rwxr-xr-x  1 dan  dan  1251 Dec  5 03:00 load_xml_into_db.pl
-rwxr-xr-x  1 dan  dan  1404 Jul  3 20:40 load_xml_into_db_git.pl
-rwxr-xr-x  1 dan  dan  1404 Jul  3 20:40 load_xml_into_db_svn.pl

Invoke load_xml_into_db.pl and it determine the type of XML and then invoke
one of load_xml_into_db_git.pl and load_xml_into_db_svn.pl

In turn, each of those scripts will use one of these modules:

[dan@devgit-ingress01:~/scripts] $ ls -l ~/modules/xml_munge*
-rwxr-xr-x  1 dan  dan   5392 Dec  5 02:51 /usr/home/dan/modules/xml_munge.pm
-rwxr-xr-x  1 dan  dan  36848 Dec  5 02:24 /usr/home/dan/modules/xml_munge_git.pm
-rwxr-xr-x  1 dan  dan  31944 Dec  5 02:37 /usr/home/dan/modules/xml_munge_svn.pm

xml_munge.pm determines the type of repository this commit came from. This
is important because there are slightly different data in the XML. For
example, subversion does not have a commit_hash.

load_xml_into_db.pl learns the type from this data in the XML file:

<UPDATES Version="1.4.0.1" Source="git">
<UPDATES Source="subversion" Version="1.3.2.2">

The Version numbers refer to the code module and is used. Yet.

Properly bring a port back from the dead.

After a port is deleted, it may be brought back.
The code was looking for that, but not detecting it
because of path issues.  Let's strip the leading /ports/head or
/ports/branches/2020Q4 and use what is left.

While here, leave the loop once we have found something to resurrect.

remove FreshPorts::Branches::GetPathToRepoForBranchCHROOT which seems
outdated for git and does not work

Add FreshPorts::Constants::GitRepos & FreshPorts::Constants::RepoLabelsToGitRepoNames

'become compatible with XML::Writer 0.900'

The example email server should be a hostname, not an IP address.

add newline to end of file

Sometimes you want 2020Q3 (usually for the repo), sometimes you want
branches/2020Q3 (usually in git)

Add in ports-quarterly

Add missing dependency: constants

This is the start of passing in the Repository name to many functions to enable correct repo-on-disk identification

remove commented out code referring to {repository} = 'ports';

Correct unkonwn to unknown

do 2>&1 to redirect stderr to std out so we capture the git stuff into the .loading log file

Line up some columns
use the right name in logging.
Use the repo we told to use.
delete commented out code

Removing blank lines.

Add two functions needed for processing git commands.

Perhaps they should be in another file, but config.sh is included by thoese
that need these.

These two have been replaced by job-waiting.pl

Mention that we only extract values from the ports repo, not the other repos

While attempting to figure out a 'wide character' issue

* https://github.com/FreshPorts/git_proc_commit/issues/

I rewrote the query for saving commit_log - this was to get
all the encoding done by $dbh->do().

It didn't help.  What heped was ;client_encoding=UTF8 on the database
connection string.

Set client_encoding=UTF8 on connect

Refer to master, not head.

We have jobs for both git and svn, and for ingress and freshports

Now that we have git and svn processing, create a separate script.

Make this work for the new repo locations and names

Fairly stable now

Remove unused $DaysRefreshed

Sort contents of %Jobs

Add $FreshPorts::Config::CheckGit => 'check_git.sh

Use commit log message not email subject

With git processing, we run off the repo, not incoming commit email.
Thus, we must change.

Remove the usr files

re: https://github.com/FreshPorts/freshports/issues/152

The nvidia drivers are needlessly marked as IGNORE. For example https://www.freshports.org/x11/nvidia-driver-390/

The message is:

IGNORE: requires kernel source files in SRC_BASE=/usr/src

This message is produced by https://svnweb.freebsd.org/ports/head/Mk/Uses/kmod.mk?view=markup around line 25

Example:

[dan@dev-ingress01:/var/db/freshports/ports-jail/var/db/repos/PORTS-head/x11/nvidia-driver-390] $ make -V IGNORE
requires kernel source files in SRC_BASE=/usr/src
The code looks for ${SRC_BASE}/sys/Makefile

Creating that file avoids the issue:

$ sudo mkdir /usr/src/sys
$ sudo touch /usr/src/sys/Makefile
$ make -V IGNORE

$

add in python_pkgnameprefix

'Bring in new PYTHON_PKGNAMEPREFIX stuff, abort on UNKNOWN branch'

extract PYTHON_PKGNAMEPREFIX from the port

Add the correct command to create the required thing

Create git branch

Remove reference to unset value.

Adding, it was never added before

Create a fake /usr/src/sys/Makefile to avoid spurious INGORE messages
by ports which use kmod

re: https://github.com/FreshPorts/freshports/issues/152

'fix a proctitle error with wide characters'

'remove debugging date'

Add missing require: FreshPorts::committer_opt_in

This will avoid this error:

nUndefined subroutine &FreshPorts::CommitterOptIn::RecordErrorDetails called at ./process_vuxml.pl line 124, <> chunk 1.\n

Uncomment code to detect debugging mode

use https, not http

We have a report on maintainer ports

Add PGSSLMODE

Add BASEDIR_PACKAGER

We don't need this any more. We use zfs rollback instead

remove system() which did nothing

'Add new new signals: * new_repo_ready_for_import * new_repo_imported'

Add new new signals:

* new_repo_ready_for_import
* new_repo_imported

whitespace only

ouch

fix

These are helper scripts, mostly for use with commits after they have
arrived. When you're looking for the raw file, use these tools

Stop using variables before testing if they are defined

This is fallout from testing for:

* https://github.com/FreshPorts/freshports/issues/133
* https://news.freshports.org/2020/01/25/the-slave-ports-for-devel-binutils-did-not-get-deleted/

'Always include the SQl when displaying an error'

Strip characters which are not valid in XML

Add much debugging output.

Always include the SQL in the error message.

'fix https://github.com/FreshPorts/freshports/issues/121'

Repair duplicate and extraneous results.

fixes https://github.com/FreshPorts/freshports/issues/121

Good script, but checkout-branch is better

Also refresh etc/make.conf in the PORT JAIL

'We have two scripts which check master-slave. One queries the databse (.pl), the other queries the Makefile (.sh). Fix both and comment appropriately'

Provide a script to copy the Jail scripts into the jail.

sysutils/bacula-server and sysutils/bacula-client are deprecated.

Use bacula9 instead

Whitespace only

Remove leading space

'Add new config.pm value: $FreshPorts::Config::ScriptDir, Start using Email::Address::UseXS because of upstream deprecation'

Add the element pathname, so we see the branch

Add punctuation to logging messages

whitespace changes

Use Email::Address::UseXS

delete existing flavor information for this port before adding

'Adopt a new approach by putting all the directory queue locations into a hash'

Adopt a new approach by putting all the directory queue locations into a
hash

'give the missing ports_categories email a better subject'

'add package flavors'

Be sure to install scripts/vars.sh.sample

Add port flavors, much whitespace adjustment while here.

Get the number of commits processed today.

Turn debugging off

Delete this, created in error

Add a script to test outgoing email

Replace p5-Mail-Sender with p5-Email-Sender

Because of this, pass in in the headers as a hash, not a string.

'honor $FreshPorts::Config::ReportDebugging'

'Allow ssl_mode to be set on database connections'

whitespace changes only

Since this is now a periodic, and it's actually running now,
make it work since we need to send an email. The output goes to a log file
via:

$ grep output /etc/periodic.conf
hourly_output=/var/log/hourly.log
daily_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"

Change the default value for FRESHPORTS_JAIL_USER to freshports, from dan

Set defaults
Correct SQL

'better process_updating.sh logging. extract CONFLICTS, CONFLICTS_BUILD & CONFLICTS_INSTALL from port'

Take a copy

Ignore config.sh

Maintain these three new database fields on the ports table

* conflicts
* conflicts_build
* conflicts_install

Better logging messages

'catch more errors if vuxml_latest.pl fails'

Return the value from CreateVulnHTML so the calling script can take action
upon error.

'When things go wrong, log it'

Add a prefix for easier logcheck filtering

Add a prefix so it's easy to filter via logcheck

When saying not found, mention not found by whom

Create branches, tags, trunk

Move

Add

Split the code into two parts:

1 - scripts run from the command line
2 - Perl modules used by some of those scripts

Split the code into two parts:

1 - scripts run from the command line
2 - Perl modules used by some of those scripts