Correct the ports tree file count

Reported by Laurent Chardon

'restrict refresh-all-active-ports.pl to head only'

'e-and-options_name.pl for jexec. add set-pkgmessage.pl never saved before'

'update refresh-all-active-ports.pl for modern times. update set-pkgmessage-and-options_name.pl for jexec. add set-pkgmessage.pl never saved before'

'add code for make -V WWW'

Update to accommodate new WWW entry in Makefiles

see https://cgit.freebsd.org/ports/commit/?id=ddd0e820c8eb73acef94c72434c382982d0fa329
Add SET CLIENT_ENCODING TO 'ISO-8859-1
to accommodate DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xe4 0x20 0x3c at ../port.pm line 406.

This still won't work for ports which use the pkg-descr for another port

e.g. https://cgit.freebsd.org/ports/tree/multimedia/linux-c7-libtheora/Makefile?id=63f36d326290febc7af8820d7bde9c78cda71c80#n17

I have emailed se@FreeBSD.org about this edge case.

Save changes from a year ago when this script was converted to use jexec

'remove maildrop/mailfilter - no longer used now we're on git'

Start using Try::Tiny

I was seeing errors on printing the SQL. Let's see if that traps that.

Testing  was not able to reproduce it.

Add more logging

Add a better log message

'Correct the link to commits

fixes https://github.com/FreshPorts/freshports/issues/335

reported by Matthew Seaman via private email
'

Correct the link to commits

fixes https://github.com/FreshPorts/freshports/issues/335

reported by Matthew Seaman via private email

'add newlines to refresh-ports.pl'

add newlines to print statements

'use correct path for pkg-message'

Correct the path to pkgmessagepath - this is not absolute, but relative

Add in long options:

dryrun y/n
debug  y/n
limit  How many rows to process
offset Where to start int the result set

Stop using syslog, because output is now logged to a file

We shall keep distfiles around - no sense removing them if we are running multiple tests

add script name to syslog call

2.1.15? Should be 2.0.15

'add in $FreshPorts::Config::PortsToRefresh'

add $FreshPorts::Config::PortsToRefresh signal

Create PORTSTOREFRESHFLAG flag

While here, derive BASEDIR_PACKAGER from BASEDIR

'move from ports-jail to jail'

Remove tag created in error

remove tag created in error

Remove variables no longer present.

use /usr/ports; /var/db/repos is now gone

Move from chroot to jexec.

The ports-tree at ~freshports/ports-jail is no longer used.
Instead, we use one in a true proper jail.

For fortnightly, do 18 days, because it's not a perfect fortnight

Move to a /usr/ports location for the jail

Move from chroot to jexec

With make-pkg-message-in.sh, move back to apply-slist : it's great

Replace JailApplySList with JailPkgMessage, for better pkg-message handling

'Fix the way we detect a commit against security/vuxml'

'check ports_categories when not processing commits'

Update to check ports_categories when not processing commits.

Add this to /usr/local/etc/freshports/config.sh:

CHECKPORTSCATEGORIESFILE="${BASEDIR}/signals/check_port_categories"

Add in some ignores with propset

Comment out the debugging code

column alignments

'fix queue-status.pl, now checks for .errors files'

Properly reresurrect a port so that our local copy stays up to date.

re: https://news.freshports.org/2021/07/04/when-a-deleted-port-is-resurrected/

Fix path manipulation with on a branch

'We don't use Jail/scripts/make-categories.sh in the port-jail'

Include make-categories.sh

chown root:wheel too

Add make-apply-slist.sh

Fix email sending.

https://cgit.freebsd.org/ports/commit/?id=3505159e72da2d8313ce2a8ddc527de700b5bc37
contains some unicode characters,

Let's use Simple::Email for that.

use spaces, not tabs to column align assignments

No real changes.

Amend some debugging output to help in the future.

sort the use statements

Remove get-list-of-current-categories.pl, not used.

Add get-list-of-current-categories.sh back in, removed via 5713

I really should have waited. Kept this only on dev for a few days.
Then test, staging, and prod would not have been affected.

Remove this, now handled by get-list-of-current-categories.pl

the new HTML from Jethro

adjust the comment headers

'use FreshPorts:: again'

'use FreshPorts::'

Seems we don't need this after all

Add missing '

Remove erronously placed tag

Try FreshPortVerify

'Create 2irTry FreshPorts::Verify'

Rename

Add

'bringing in better category detection'

Be cautious when printing svn_revision - I think this arose when writing
Verify/set-pkgmessage-and-options_name.pl

Get the list of categories from the repo

FreshPorts::Config::scriptpath and
FreshPorts::Config::ScriptDir have the same value. use only the latter.

Also, update options_name

Bring in the changes from https://github.com/FreshPorts/freshports/pull/261

'Previous release was mucked up. updated config.sh.sample'

adjust from svn to git

Add the make-apply-slist.sh code

'if no pkg-message in the repo, try apply-slist'

'Updated config.sh.sample'

Not every directory in the root directory of the repo is a category. This
script pulls back a list of those categories.

This need arose with this commit:

https://cgit.freebsd.org/ports/commit/?id=bbc2474ef7a65eb8561c8ecf7af80c2bfed1f248

Which created the .git directory.

'use main, not master, for head'

stop counting .txt & start counting .xml

'add DBUSER_REPORTING to config.sh.sample'

'add options_name'

'freebsd-cvs.sh: say if config.sh is not found
queue-status.pl: stop using the database, sort the queue keys so Nagios output is consistent, look only for files > 5 minutes old.
config.sh.sample: mention PORTSDIRBASE is best without a trailing slash
Add the monitoring directory, but I am not sure it is used yet.
process_vuxml.pl: improve spacing'

cate for OPTIONS_NAME

'adding committer_name, committer_email, author_name, author_email'

'bring in the vuxml code from trunk; take author and committer in.'

Bring in the vuxml code from trunk

Pull in the vuxml code from the main branch, because we need the new vuxml
code here too.  While here, do some betting debugging.

'In /var/db/ingress/message-queues/incoming, we look for *.xml'

'better queue monitoring'

'add virtualhost-common-ssl.conf.sample'

queue-status.pl no longer uses FreshPorts::Utilities::CommitCountPeriod

'conver all the queues'

'add more queue monitoring'

Add ingress_svn/message-queues/incoming & ingress_svn/message-queues/spooling to the queue checking

PORTSDIRBASE is best without a trailing /

Convert for use by git processing.

Our files are now *.txt not *.txt.raw

There was no need to rename them

Show reasons for update and number of updates

'Adjust process_vuxml.pl to work on file open'

Change to allow IO::String on incoming new()

'For new security/vuxml/vuln.xml layout - perhaps 1.42 was a duplicate tag - oops'

Swap from vuxml_parsing_via_dom to vuxml_parsing

If fetching (svn up) fails, log a fatal error.

The rest is just whitespace changes

Remove empty error files
Add more debugging output for Branch and revision

When invoking parse, specify ProtocolEncoding => 'ISO-8859-1'

That avoid errors like this:

not well-formed (invalid token) at line 28, column 23, byte 1305 at /usr/local/lib/perl5/site_perl/mach/5.32/XML/Parser.pm line 187.

on this:

Reported by Andr?? Bargull on 2020-11-2</li>

see vid="01ffd06a-36ed-11eb-b655-3065ec8fd3ec"

Let's save this before deleting it.

Test worked, now moving code to process_vuxml.pl

remove FreshPorts::Constants::SVN_Repo_Dir_Name_Ports which is not used

Fix up CanWeProcessThisBranch() was was passing an unnecesary parameter
because it should have been invoking GetPathToRepoForBranchSVN() not
GetPathToRepoForBranch()

'first cut of git modules'

branches.pm

* add GetPathToRepoForBranchSVN
* for a given branch name, return the full path.  Including the CHROOT path.

constants.pm

* add FreshPorts::Constants::SVN_Repo_Dir_Name_Ports = ports-HEAD
* is that used?

port.pm

* make this script work with both git and svn
* the repos are in different locations
* remove 

process_svn_mail.pm

* Amend XML to include the source (subversion) so that XML processing knows which code to invoke
* When invoking XML::Writer, don't specify OUTPUT, becuase that borks the code

verifyport.pm

* start passing around RepoType so everyone knows if this is git or svn

xml_munge.pm

* remove a bunch of commented out code now that most of this processing is in one of  xml_munge_git.pm & xml_munge_svn.pm
* we open the XML, check git or svn, and invoke the respective code to complete the processing

xml_munge_git.pm
xml_munge_svn.pm

* adjust the code so that non-ports don't start compiling lists of ports to save
* in xml_munge_git.pm, remove the code which checked the length of message_id to determin git or svn
* that detemination is carried out by xml_munge.pm and it checks the XML to decide that

load_xml_into_db.pl

* modified to check the source repo for this XML
* then invoke one of FreshPorts::xml_munge_git or FreshPorts::xml_munge_svn

load_xml_into_db_git.pl
load_xml_into_db_svn.pl

Modified to invoke the approviaate XML munging class:

* my $Munger = FreshPorts::XML_Munge_git->new($dbh);
* my $Munger = FreshPorts::XML_Munge_svn->new($dbh);

I started to modify these scripts to work with git, then realized they are
no longer used.

email and commit processing is handled by services:

[dan@devgit-ingress01:/usr/local/libexec/freshports-service] $ ls -l
total 26
-r-xr-xr-x  1 root  wheel  7267 Dec  6 22:17 freshports.sh
-r-xr-xr-x  1 root  wheel  1387 Dec  6 20:16 ingress.sh
-r-xr-xr-x  1 root  wheel  4109 Dec  6 22:13 ingress_svn.sh

Look there for invoking the xml processing scripts

We don't use this any more

Convert from `stuff` to $(stuff)

Convert from `stuff` to $(stuff)

Trim down the code by removing unnecessary items, such as the database
connection.

Use the right value for branch name.

Let's just do a fetch, not a pull

Convert the XML repo label to a database repo name

Patch to work with XML-Writer-0.900

Without this patch, the error is:

Output must be a handle, a reference or 'self' at /usr/local/lib/perl5/site_perl/XML/Writer.pm line 480, <> line 114.

This ties in with https://metacpan.org/changes/distribution/XML-Writer for
0.900:

"Fail when a scalar other than 'self' is passed as OUTPUT."

Fix the branch

spacing

spell quarterly

Add a test for showing repos:

[dan@devgit-ingress01:~/modules] $ echo perl ./Testing/GetRepos.pl| sudo su -fm freshports
ports has REPODIR='/var/db/freshports/ports-jail/var/db/repos/freebsd-ports' and REPODIR_CHROOT='/var/db/repos/freebsd-ports'
ports-quarterly has REPODIR='/var/db/freshports/ports-jail/var/db/repos/freebsd-ports-quarterly' and REPODIR_CHROOT='/var/db/repos/freebsd-ports-quarterly'

Change GetRepoNameForBranch() to use newly FreshPorts::Constants::GitRepos

We want to convert ports and ports-quarterly to freebsd-port and
freebsd-ports-quarterly respectively. We have labels; we want directories.

remove commented out code with hardcoded commit_hash, repo, and FileRevision

put a space before: for git

Document where we need to delete the port

How did that = get in there?

Correct the name of the script

Some colum alignment

Adjust whitespace

Put a tag in there, so we know where it's coming from

Add SET CLIENT_ENCODING TO 'SQL_ASCII' on a select.
Not everything is UTF-8, so we have to do this for now.

Whitespace some stuff

Moving to git, we take a slightly different approach.  We need multiple
working copies because there are multiple git repos:

* freebsd
* freebsd-ports
* freebsd-docs

Change path_to_tree to RepoDir
Move CheckGit from freshports to ingress signal directory

Save commit_hash_short

We renamed SVNBaseDir

Add the script which checks out a specific commit

Add deails for the ingress user directories.

Add other details pertaining to git processing

Add in the git repos

While here, column align some stuff

Sort the jobs

Spell: element id

'rm /usr/src/sys/Makefile from the ports-jail'

'Bring in new PYTHON_PKGNAMEPREFIX code - for real this time'

Add in make-flavors-package-names.sh

'Bring in new PYTHON_PKGNAMEPREFIX code'

Combine some lines into one.  Whitespace only.

Some simple updates

Create branch for git work

Need this to fix IGNORE for ports which use kmod

re: https://github.com/FreshPorts/freshports/issues/152

'Create a fake /usr/src/sys/Makefile to avoid spurious INGORE messages
by ports which use kmod
'

'convert all http to https, get ready for MaintainerPorts'

Avoid readproctitle errors by doing UTF-8

remove hardcoded date used for debugging

'add report-maintainer-ports.pl'

Go case insensitive on the email address match

remove:

* NotificationNonPorts & NotificationDirectories - they appear to be unused
* add MaintainerPorts for new report

'add PGSSLMODE to config.sh'

'add BASEDIR_PACKAGER to config.sh'

'remove system() which did nothing in process_vuxml.pl'

Save recent work, before removing.

first version, going to start using -e next

'job-waiting.pl now handles the new packages code'

Change our job-waiting loop.

When the job_waiting signal is set, run the waiting jobs and keep
runnint them until we don't find anything more to run.

fix

fix

clean

Damn.  That was wrong.

'Don't ignore everything after Directory Properties'

In the original version of this code, we always ignored everything after a
Directory Properties line.  We now just ignore that section.

This error led to whole swaths of emails being ignored.  Oops.

see https://github.com/FreshPorts/freshports/issues/133
    https://news.freshports.org/2020/01/25/the-slave-ports-for-devel-binutils-did-not-get-deleted/

'fix broken XML chars'

Note that FreshPorts::Config::email_server must be a hostname, not an IP
address.

Always include the SQl when displaying an error

remove extraneous /

'minor changes, no bug fixes'

Use $0 instead of hardcoding our own script name

Align the echo and the code to make it easier to read.  Whitespace only.

We have two scripts which check master-slave.

One queries the databse (.pl), the other queries the Makefile (.sh).

'update test-master-port.sh now that sysutils/bacula-server is gone from the tree'

'fix hard-coded hostname'

Stop using Email::Address::UseXS because things have been fixed. remove commented out code.

'* Add element_pathnane to missing-port-categories.sh so we see branch details'

Check out the given branch to our local repo directory

When we can't find the branch for a commit, try checking it out.

Add FreshPorts::Config::ScriptDir for use by perl modules which need to run
scripts.

Add hostname to test email

'remember to delete package_flavors before adding'

Remove vertical whitespace to avoid:

replaced vertical whitespace in Subject header with space; this will become fatal in a future version at /usr/local/lib/perl5/site_pe

'removing status.pm'

Give the email subject a correct description

'add Flavors, adjust ccommitter_opt_in to use new email modules, assign default value to database.pm::sslmode, update email.pm to use Email::Simple not Mail::Sender(now deprecated)'

Add make-flavors-package-names.sh script

'add package flavors'

Add a way to refresh just one port. You must modify the code to specify that
port

Add a script to get:

* Number of queued commits
* Commit processing status

'Convert from using p5-Mail-Sender to p5-Email-Sender, Honor $FreshPorts::Config::ReportDebugging for reports, Add test-sending-email.pl'

'Replace p5-Mail-Sender with p5-Email-Sender'

Convert from using p5-Mail-Sender to p5-Email-Sender

* headers are now passed in via a hash

report-new-ports.pl now takes a range, as originally intended

show-config.pl: stop using email_client and use only email_server

Remove email_client: please use email_server instead

Add ReportDebugging, to allow reports to run without sending anything.

Honor $FreshPorts::Config::ReportDebugging and stop using $Debug

''

'better defaults in config.sh.sample, make missing-port-categories.sh email results'

Set default ADMINEMAIL to me

Set sslmode in config.pm

'-m'

Add

When inserting, set a value for the commit_log_ports.port_name_revision
field. This is used when comparing against the ports.conflicts* fields.

Extract three new values from the Makefile:

* CONFLICTS
* CONFLICTS_BUILD
* CONFLICTS_INSTALL

'Fix vuxml non running, improve logging'

All of the following deficiencies led to hiding permissions issues on a
directory.

* mention when process_vuxml.pl begins
* say vuxml_ident.pl, not vuxml ident
* say vuln_latest.pl, not vuxml latest
* toss the output of vuln_latest.pl into the log file
* if an error occurs during vuln_latest.pl, refer the reader to the logfile

Instead, now we see this in the vuxml.log file:

###
process_vuxml.pl finishes
'/var/db/freshports/cache/html' exists
trying to open '/var/db/freshports/cache/html/vuln-latest.html.tmp'
could not open '/var/db/freshports/cache/html/vuln-latest.html.tmp'
Tue Aug 14 14:59:55 UTC 2018 process_vuxml.sh vuxml finishes
####

'fix broken vuxml running'

There is not constants
Only FreshPorts::constants

This was causing vuxml to not be processed.

There is no FreshPorts::warnings.
Only warnings

When things go wrong, log it.

When saying not found, say by whom

Oops.  Didn't mean to do exactly that.

Rename to trunk

Rename, while I create branches, tags, trunk

Move scripts into a new scipts directory.
Create new modules directory

Code for processing incoming commits