'fix ports being deleted on branches'

verifyport.pm: fix ports being deleted on branches

re: https://github.com/FreshPorts/freshports/issues/528

If the port is deleted, but set active, set it deleted.
After saving the port, make sure we adjust the {status} flag:

If the status changes, set it back to what it was

see https://news.freshports.org/2024/11/27/when-a-port-on-a-branch-is-deleted-freshports-tried-and-fails-to-refresh-it/

Rename upate_generate_plist()
    to update_generate_plist()

process_vuxml.pl: When the parsing has finished, say so.

re: https://github.com/libexpat/libexpat/issues/928

Also, dispose() of the doc when completed.

'Slave ports need to have port vulnerabilities adjusted after a commit to the master port'

After processing the commit, be sure to adjust the vulnerability count on
any slave ports.

https://github.com/FreshPorts/freshports/issues/607

This is already done for each port in the commit, but slave ports were
previously missed.

column alignments

'missing-port-categories.sh - make it OFFLINE aware, change get-list-of-current-categories.sh so it no longer observes OFFLINE.'

With recent changes to config.pm.sample:

* Stop using $FreshPorts::Config::OFFLINE
* start using $FreshPorts::Config::Offline instead

Remove $FreshPorts::Config::OFFLINE - which had an incorrect path
   Add $FreshPorts::Config::Offline - which has the correct path

Do not run if OFFLINE

saving incompletely tested stuff, as a checkpoint for new categories

'new OFFLINE code, fix new categories stuff, UPDATING will now work with or without a space
'

Fix category creation:

Fix the order of parameters on JailCategoryDescrptionScript

No need to fetch the category Makefile - it's already there thanks to git

re: https://news.freshports.org/2024/11/07/fixing-the-category-creation-code/

refer to $0, not the incorrect test-master-port.sh left over from copy/paste

kxml_munge_git.pm: specify the correct branch for Sanity Test Failures

We were specifying whahtever branch we found in the xml file. Instead, now
we show the converted branch name ('main' from FreeBSD, is 'head' in
FreshPorts-land.

also, only show the branch on the URL within the message if it is not head.

Create and start using:

$FreshPorts::Config::OFFLINE

This replaces:
$FreshPorts::Config::ScriptDir/OFFLINE

'RIP bacula9, long live bacula15 - this time for the shell script'

'RIP bacula9, long live bacula15'

On sanity test messages, supply the branch

re:https://github.com/FreshPorts/freshports/issues/564

'Add host URL to newusers.pl'

'dump-package_imports.sh: allow for sorting by date and name (default)'

'dump-package_imports.sh: add'

dump-package_imports.sh: for nulls, sort by name, package_set

add notes about the environment variables

Adjust for new package importing strategy

Add:

* FreshPorts::Config::LogDir
* FreshPorts::Config::LogJobsWaiting

Remove:

* $FreshPorts::Config::NewRepoImported

Adjust for new package importing strategy

see: https://github.com/FreshPorts/packages-import/releases/tag/2.7.0

job-waiting.pl

* remove references to FreshPorts::Config::NewRepoImported
* start logging to FreshPorts::Config::LogJobsWaiting
* no need to launch after report-notification-packages.pl after UpdatePackagesFromRawPackages.py
* that is done by import-via-copy-packagesite-all-raw-fields.py
* wow, in hindsight, I should have created a wrapper script
  which invokes import-via-copy-packagesite-all-raw-fields.py, and
  UpdatePackagesFromRawPackages.py, and then report-notification-packages.pl
* Perhaps later.

report-notification-packages.pl

* If there is nothing to report upon, don't error out
* no need to remove NewRepoImported any more

job-waiting.pl - fix typo: Beacuse to Because

database.pm: Add sslcertmode=disable

Fix this issue:

could not open certificate file “/root/.postgresql/postgresql.crt”: Permission denied

I’m positive the problem is the connection is asking for a client certificate.
I checked my pg_hba.conf configuration and found no lines mentioning cert.

See also https://github.com/FreshPorts/packages-import/commit/10285544ad72311a82ed36cf9153a41db4722439

'Standardize all report email subjects'

'loop detections added to job-waiting.pl; missing-port-categories.sh will log better; add in report-notification-packages.pl'

Attempt to standardize on one name for emails:

FreshPorts notification

Start keeping stats
Explain what update, delete, insert will mean.

Attempt to standardize on one name for emails:

FreshPorts notification

This now properly handles package_set

Perhaps a loop

Change the rm to rm -f when removing the flag

This script is invoked by a periodic script. The FLAG won't be present then.

use nginx and fp_listen - modernize

Created in error. All wrong.

refresh-abi.sh: alter logging details

Add process id

'remove some constants, improve abi refresh'

No  need to touch FreshPorts::Config::RefreshCachFileFlag now

re: https://github.com/FreshPorts/freshports/issues/58

Let's not output anything by default. Let there be less.

Renove RefreshCachFileFlag - no longer used

re: https://github.com/FreshPorts/freshports/issues/58

While here, remove some items from show-config.pl which are no longer
present.

* DynamicDir
* FlagDir
* RefreshCachFileFlag
* WWWENPortsCategoriesFlag

improve logging for gitCheckoutResults

column aligment

add comment details

'config.sh - add ABILOG; refresh-abi.sh - better logging'

add ABILOG

Remove this tag - going to replace it

Update PostgreSQL items

* For use by new abi scripts:
- Use export on PGSSLMODE
- Add PGSSLROOTCERT
- add DBUSER_ABI

add commit, and logging

add trailing ; to sql

Maintain the list of ABI

ABI get added, deleted. This set of scripts will maintain
that list in our abi table.

remove some blank lines

Igmore git-last-processed-tags.sh

Better comments

Use ne not != for comparison

Update to reflect we are using actual jails now, not just chroot.

Use the full path for git

Things changed on some servers with FreeBSD 14.0

remove this tag, recreated/overwitten in error

'removing our flag when we start processing. tell port->save() we are working on HEAD'

mention that process_default_versions.pl operates only upon head

On port->save(), specify HEAD, because that is only what this should be invoked for

refer to $0, not process_moved.sh

add $FreshPorts::Config::CheckPortsCategoriesFlag back in, and sort the flags

Add utf8, DefaultVersionsFlag, and remove CheckPortsCategoriesFlag & WWWENPortsCategoriesFlag

job-waiting.pl
* remove FreshPorts::Config::WWWENPortsCategoriesFlag => 'process_www_en_ports_categories.sh
* add FreshPorts::Config::DefaultVersionsFlag      => 'process_default_versions.pl

remove:

* process_CVSROOT_approvers.sh
* process_www_en_ports_categories.sh

add process_default_versions.pl

'process Mk/bsd.default-versions.mk results'

'Cater for broken 2023Q4 branch'

'add categories_rebuild_search_html.sh'

'add categories_rebuild_search_html.sh'

'Include changes adding port_id to RemovePortFromCache() calls'

'Include changes adding port_id to RemovePortFromCache() calls'

'do better cache invalidation for MOVED, UPDATING, etc'

Caching changes

12345678901234567890123456789012345678901234567890123456789012345678901234567890
Items like MOVED & UPDATING where never cleared from cache. caching.pm was
designed when the web server and the ingress server where always assumed to be
on the same host. That is no longer the cache. The code has been rewritten to
use the same cache clearing mechanism as in other parts of the system.

A new table has been created: cache_clearing_files

It functions much like the existing cache_clearing_ports table

The changes here are coordinated with changes to the modules port and to the
fp-listen (part of the webserver) code.

re: https://github.com/FreshPorts/freshports/issues/468

spell with

Use this shebang: #!/usr/local/bin/perl -w

Actually join the tables in this quewry

'do not pass empty strings to _GetRealPath()'

Use case insensitive matches for Affects and Author

This allows for:

20230213:
  Affects: users of sysutils/nut*
  AUTHOR: cy@FreeBSD.org

Whereas most people do upper case:

20230213:
  AFFECTS: users of security/logcheck
  AUTHOR: yasu@FreeBSD.org

'-m'

Make sure we use head not branches/main

Add FreshPorts::Constants::MAIN as a constant
and when checking to see what branch we are on,
convert main to head, because the XML gives us
main, but we are using head in the database.

fixes #435

Remove some old subversion specific code.

FetchAllFiles()
svnUpFile()
xml_munge_svn.pm

Thank you.

Use freshports, not FreshPorts, in URLs.

see: https://github.com/FreshPorts/freshports/issues/412

Only update if test_depends differs

'add TEST_DEPENDS'

Extract TEST_DEPENDS

re https://github.com/FreshPorts/freshports/issues/411

'refresh-ports.sh & process_updating.sh now log to DIRLOG, process_updating.pl looks for AFFECTS and avoids duplicates in cache_clearing_ports, add show-config.sh'

Be clearer about how this directory is used.

Populate ports.makefile field re: https://github.com/FreshPorts/freshports/issues/406

Fix vuxml processing. The files have moved from security/vuxml/vuln-YYYY.xml to
security/vuxml/vuln/YYYY.xml

Fix the regex for checking for such commits. Add more logging.

More detail in https://github.com/FreshPorts/freshports/issues/360

Remove code added for debugging

Add

Log the output of process_updating.pl to process_updating.log

'remove dir-create.sh; correct ports_tree_file_count.pl'

This script is deprecated.

Remove tag created in error

Restrict this port to head

'add code for make -V WWW'

Add WWW

Save changes from a year ago when this script was converted to use jexec

Adjust SQL to work only on head.

Adjust Port->new() call to specific 'git' repo

'try catching sql print errors'

Better logging message

Drop the procmail, actually maildrop filter

Since FreeBSD moved to git, this is no longer required.

'Avoid periods at the end of URLs and email addresses'

Avoid periods at the end of URLs and email addresses

column alignment

go for 2500 row sets

Be sure to prepare the repo with the latest data before processing

'add launch-refreshes.sh & long options for refresh-ports.pl'

A script for launching multiple instances of refresh-ports.pl which each run
in parallel

log to a file

'add script name to syslog call in refresh-ports.pl'

'add in $FreshPorts::Config::PortsToRefresh'

'add $FreshPorts::Config::PortsToRefresh, refresh-ports.sh, refresh-ports.pl'

we shall assume git processing when refreshing

Add the ability to process the ports_to_refresh table.

fix comment typo

'move from ports-jail to jail'

'move from ports-jail to jail'

'move from ports-jail to jail'

fix typo in /usr/ports comment

Stop using ports-jails, start using a proper jail

Use /usr/ports within that jail

this is deprecated

We are now using a FreeBSD jail for port data extraction

Move the jail to /usr/ports, not /var/db/repos/ports

Better pkg-message handling, but not great.

Better pkg-message handling, but still not great.

Fix the way we detect a commit against security/vuxml

Look for any file like vuln-YYYY

'check ports_categories when not processing commits'

Update to check ports_categories when not processing commits.

Add this this to /usr/local/etc/freshports/config.pm:

$FreshPorts::Config::CheckPortsCategoriesFlag   = $FreshPorts::Config::BaseDir . '/signals/check_port_categories';

Script for testing categories obtained from the repo

First attempt at keeping track of what reports have not been run

column alignments

'Avoid variables without values in verifyport.pm
Add a proper Undelete() for ports in port.pm
Fix paths for branch commits in xml_munge_git.pm
'

Report upon .errors files
Don't report upon ingress-svn any more

reduce the number of error messages we get in the the logs by ensuring a default value exists

make-apply-slist.sh is not required in the port jail.

This functionality is carried out by get-list-of-current-categories.sh

Better shorter way to get the list of categories

'Use Email::Stuffer'

'add error handling for categories.pm'

Detect and raise a fatal error if the script fails

Print the categories

'add get-list-of-current-categories.sh back in'

'-m'

Commit a work in progress, only to be deleted on my next commit

Remove 'svn commit checking status' since we don't do subversion commits any
more

Document that FreshPorts::Config::FreshPortsURL must have a trailing /

Use FreshPorts again...

Use FreshPorts::

'fix Makefile.PL'

'Create FreshPortsVerify'

'Create FreshPortsVerify'

Move

Move from FreshPorts::Scripts -> FreshPorts::Modules

'Create 1irAdd Makefile.pl'

'Create first tag'

'bringing in better category detection'

Stop using IgnoredItems and use a list of Categories pulled from the repo

Add categories.pm used thusly:

use FreshPorts::categories
use List::MoreUtils 'any';

FreshPorts::categories::FetchAll();

# look for special files outside a port, such as LEGAL, GIDs, UIDs, .hooks
if ( ! any {/$category_name/} @FreshPorts::Categories::categories ) {
...
}

With IgnoredItems being removed, this script is no longer required.

FreshPorts::Config::scriptpath and
FreshPorts::Config::ScriptDir have the same value. use only the latter.

Fix the debug output so it makes sense when reading it.

'add get-list-of-current-categories.sh; fix vuln_latest.pl for new HTML usage'

Commit my changes to vuln_latest.pl before incorporating changes from
https://github.com/FreshPorts/freshports/pull/261

'if no pkg-message in the repo, try apply-slist - corrected'

space changes only

add make-apply-slist.sh

'Previous release was the wrong tarball name - updated config.sh.sample'

A new iteration of the find

re https://github.com/FreshPorts/freshports/issues/245

Not all ports with a pkg-message have a pkg-message file.

Some have files/pkg-message.in, for example.

This script uses apply-slist to build that file.

'count *.xml not *.txt'

Now that the FreeBSD git repo is out, it seems they changed the name of the
head branch from master to main. The main change there is:

-my %BranchConversions = ( 'master' => 'head' );
+my %BranchConversions = ( 'main' => 'head' );

Adjust some constants to allow commits to process.

$FreshPorts::Config::RepoDir . '/PORTS-head'; was renamed to 
$FreshPorts::Config::RepoDir . '/' . $FreshPorts::Config::Repo_PORTS;

This meant that $FreshPorts::Config::Repo_PORTS had to be moved higher
up withint config.pm

Adjust the constants for the repo directories. We have abandoned github
and moved to git.freebsd.org.  e.g. freebsd-doc becomes doc

add DBUSER_REPORTING

'Add OPTIONS_NAME to scripts.
fix ports_tree_file_count.pl to work with new configuration items'

FreshPorts::Constants::Repo_Ports is gone.

Use FreshPorts::Constants::Repo_Dir_Name_Ports instead

Start saving options_name

Add new committer and author fields:

* committer_name
* committer_email
* author_name
* author_email

Add p5-Email-Address-XS>0:mail/p5-Email-Address-XS as new dependecy because
we need to split the user off the address.

We now take in Author and Committer.

Author is not yet written to the database.

'bring in the vuxml code from trunk'

Remove use of database

Sort the keys before displaying them so every host has the same order of
output.

In /var/db/ingress/message-queues/incoming, we look for *.xml

Remove use of database

use -mindepth 1 to avoid counting the directory itself

'K  IW   svn-commit.tmp (Modified)                                                                                                                                                                                                                      Row 3    Col 1   
Stop using FreshPorts::Utilities::CommitCountPeriod because we are counting
only file more than 5 minutes old.'

Stop using FreshPorts::Utilities::CommitCountPeriod because we are counting
only file more than 5 minutes old.

adjust report_non_zero to cover all incoming and spooling directories. same for CountRecent

trailing / issues, add in src-stable-12

Check for and error out if config.sh is not found

'fix system_status.sh'

'email files are *.txt not, not *.txt.raw'

'Show reasons for update and number of updates'

'allow IO::String on incoming new()'

Add three new paramaters:

* dryrun        - do not update the database
* showchecksums - show the checksums
* printnodes    - print the node we are processing

Change from IO::File to IO::String for the $nodeString = $node->toString();
XML. This means the fh->open($nodeString) succeeds.

'For new security/vuxml/vuln.xml layout'

'For new security/vuxml/vuln.xml layout'

Adjust to XML processing, not XML creation.
XML will be created by another user.

Change to fixed length wait times.
We are going to very high values for Fetch_Retry_Limit because svn is lagged
on replication. It is easier to wait than to error out and fix by hand.

Move to using XML::DOM::XPath now that security/vuxml/vuln.xml is split into
several files. Proof of concept courtesy of bapt@FreeBSD.org

Saving before deleting.  Work will be done by vuxml_parsing.pm

'update to 2.0.1'

add a comment showing an example make-port.sh call

add in stuff for the quarterly ports, and change tabs to spaces

'first cut of the scripts for git'

adjust spaces to tabs

Remove files no longer used.

Convert from `stuff` to $(stuff)

slightly different comments

Convert from `stuff` to $(stuff)

Revert last commit. It seems we need some of that code afterall.

Now that I've realized the website should process both svn and git commits,
I wanted a way to inspect an XML file and determine which type of repo it
came from. This is vital because we have two different sets of code to deal
with each origin:

[dan@devgit-ingress01:~/scripts] $ ls -l load_xml_into_db*
-rwxr-xr-x  1 dan  dan  1251 Dec  5 03:00 load_xml_into_db.pl
-rwxr-xr-x  1 dan  dan  1404 Jul  3 20:40 load_xml_into_db_git.pl
-rwxr-xr-x  1 dan  dan  1404 Jul  3 20:40 load_xml_into_db_svn.pl

Invoke load_xml_into_db.pl and it determine the type of XML and then invoke
one of load_xml_into_db_git.pl and load_xml_into_db_svn.pl

In turn, each of those scripts will use one of these modules:

[dan@devgit-ingress01:~/scripts] $ ls -l ~/modules/xml_munge*
-rwxr-xr-x  1 dan  dan   5392 Dec  5 02:51 /usr/home/dan/modules/xml_munge.pm
-rwxr-xr-x  1 dan  dan  36848 Dec  5 02:24 /usr/home/dan/modules/xml_munge_git.pm
-rwxr-xr-x  1 dan  dan  31944 Dec  5 02:37 /usr/home/dan/modules/xml_munge_svn.pm

xml_munge.pm determines the type of repository this commit came from. This
is important because there are slightly different data in the XML. For
example, subversion does not have a commit_hash.

load_xml_into_db.pl learns the type from this data in the XML file:

<UPDATES Version="1.4.0.1" Source="git">
<UPDATES Source="subversion" Version="1.3.2.2">

The Version numbers refer to the code module and is used. Yet.

Properly bring a port back from the dead.

After a port is deleted, it may be brought back.
The code was looking for that, but not detecting it
because of path issues.  Let's strip the leading /ports/head or
/ports/branches/2020Q4 and use what is left.

While here, leave the loop once we have found something to resurrect.

remove FreshPorts::Branches::GetPathToRepoForBranchCHROOT which seems
outdated for git and does not work

Add FreshPorts::Constants::GitRepos & FreshPorts::Constants::RepoLabelsToGitRepoNames

'become compatible with XML::Writer 0.900'

The example email server should be a hostname, not an IP address.

add newline to end of file

Sometimes you want 2020Q3 (usually for the repo), sometimes you want
branches/2020Q3 (usually in git)

Add in ports-quarterly

Add missing dependency: constants

This is the start of passing in the Repository name to many functions to enable correct repo-on-disk identification

remove commented out code referring to {repository} = 'ports';

Correct unkonwn to unknown

do 2>&1 to redirect stderr to std out so we capture the git stuff into the .loading log file

Line up some columns
use the right name in logging.
Use the repo we told to use.
delete commented out code

Removing blank lines.

Add two functions needed for processing git commands.

Perhaps they should be in another file, but config.sh is included by thoese
that need these.

These two have been replaced by job-waiting.pl

Mention that we only extract values from the ports repo, not the other repos

While attempting to figure out a 'wide character' issue

* https://github.com/FreshPorts/git_proc_commit/issues/

I rewrote the query for saving commit_log - this was to get
all the encoding done by $dbh->do().

It didn't help.  What heped was ;client_encoding=UTF8 on the database
connection string.

Set client_encoding=UTF8 on connect

Refer to master, not head.

We have jobs for both git and svn, and for ingress and freshports

Now that we have git and svn processing, create a separate script.

Make this work for the new repo locations and names

Fairly stable now

Remove unused $DaysRefreshed

Sort contents of %Jobs

Add $FreshPorts::Config::CheckGit => 'check_git.sh

Use commit log message not email subject

With git processing, we run off the repo, not incoming commit email.
Thus, we must change.

Remove the usr files

re: https://github.com/FreshPorts/freshports/issues/152

The nvidia drivers are needlessly marked as IGNORE. For example https://www.freshports.org/x11/nvidia-driver-390/

The message is:

IGNORE: requires kernel source files in SRC_BASE=/usr/src

This message is produced by https://svnweb.freebsd.org/ports/head/Mk/Uses/kmod.mk?view=markup around line 25

Example:

[dan@dev-ingress01:/var/db/freshports/ports-jail/var/db/repos/PORTS-head/x11/nvidia-driver-390] $ make -V IGNORE
requires kernel source files in SRC_BASE=/usr/src
The code looks for ${SRC_BASE}/sys/Makefile

Creating that file avoids the issue:

$ sudo mkdir /usr/src/sys
$ sudo touch /usr/src/sys/Makefile
$ make -V IGNORE

$

add in python_pkgnameprefix

'Bring in new PYTHON_PKGNAMEPREFIX stuff, abort on UNKNOWN branch'

extract PYTHON_PKGNAMEPREFIX from the port

Add the correct command to create the required thing

Create git branch

Remove reference to unset value.

Adding, it was never added before

Create a fake /usr/src/sys/Makefile to avoid spurious INGORE messages
by ports which use kmod

re: https://github.com/FreshPorts/freshports/issues/152

'fix a proctitle error with wide characters'

'remove debugging date'

Add missing require: FreshPorts::committer_opt_in

This will avoid this error:

nUndefined subroutine &FreshPorts::CommitterOptIn::RecordErrorDetails called at ./process_vuxml.pl line 124, <> chunk 1.\n

Uncomment code to detect debugging mode

use https, not http

We have a report on maintainer ports

Add PGSSLMODE

Add BASEDIR_PACKAGER

We don't need this any more. We use zfs rollback instead

remove system() which did nothing

'Add new new signals: * new_repo_ready_for_import * new_repo_imported'

Add new new signals:

* new_repo_ready_for_import
* new_repo_imported

whitespace only

ouch

fix

These are helper scripts, mostly for use with commits after they have
arrived. When you're looking for the raw file, use these tools

Stop using variables before testing if they are defined

This is fallout from testing for:

* https://github.com/FreshPorts/freshports/issues/133
* https://news.freshports.org/2020/01/25/the-slave-ports-for-devel-binutils-did-not-get-deleted/

'Always include the SQl when displaying an error'

Strip characters which are not valid in XML

Add much debugging output.

Always include the SQL in the error message.

'fix https://github.com/FreshPorts/freshports/issues/121'

Repair duplicate and extraneous results.

fixes https://github.com/FreshPorts/freshports/issues/121

Good script, but checkout-branch is better

Also refresh etc/make.conf in the PORT JAIL

'We have two scripts which check master-slave. One queries the databse (.pl), the other queries the Makefile (.sh). Fix both and comment appropriately'

Provide a script to copy the Jail scripts into the jail.

sysutils/bacula-server and sysutils/bacula-client are deprecated.

Use bacula9 instead

Whitespace only

Remove leading space

'Add new config.pm value: $FreshPorts::Config::ScriptDir, Start using Email::Address::UseXS because of upstream deprecation'

Add the element pathname, so we see the branch

Add punctuation to logging messages

whitespace changes

Use Email::Address::UseXS

delete existing flavor information for this port before adding

'Adopt a new approach by putting all the directory queue locations into a hash'

Adopt a new approach by putting all the directory queue locations into a
hash

'give the missing ports_categories email a better subject'

'add package flavors'

Be sure to install scripts/vars.sh.sample

Add port flavors, much whitespace adjustment while here.

Get the number of commits processed today.

Turn debugging off

Delete this, created in error

Add a script to test outgoing email

Replace p5-Mail-Sender with p5-Email-Sender

Because of this, pass in in the headers as a hash, not a string.

'honor $FreshPorts::Config::ReportDebugging'

'Allow ssl_mode to be set on database connections'

whitespace changes only

Since this is now a periodic, and it's actually running now,
make it work since we need to send an email. The output goes to a log file
via:

$ grep output /etc/periodic.conf
hourly_output=/var/log/hourly.log
daily_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"

Change the default value for FRESHPORTS_JAIL_USER to freshports, from dan

Set defaults
Correct SQL

'better process_updating.sh logging. extract CONFLICTS, CONFLICTS_BUILD & CONFLICTS_INSTALL from port'

Take a copy

Ignore config.sh

Maintain these three new database fields on the ports table

* conflicts
* conflicts_build
* conflicts_install

Better logging messages

'catch more errors if vuxml_latest.pl fails'