This code no longer used. see svn+ssh://svnusers@svn.int.unixathome.org/freshports-1/ingress instead Split the code into two parts: 1 - scripts run from the command line 2 - Perl modules used by some of those scripts
Do a better logging attempt
remove extraneous declaration of $FreshPorts::Config::JailBaseDir
Whitespace only
use /usr/local/bin/perl
String comparison is eq, not ==
Sys::Syslog, not Syslog
Remove FreshPorts:: from DBI references, erroneously added.
Prepend all use and require with FreshPorts::
Add this, used when building FreshPorts-Modules
If the file is blank, return from _GetRealPath
Remove trailing whitespace from a couple of lines
Fix the splitting of path:dir[:target] into dir:[:target] re https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/makefile-depend.html With help from https://gist.github.com/ktracer
Reduce the scope for $fh Revert declaration of $p to orignal location
Remove spurious committer_opt_in include introduced in r4906
Although I think this script is no longer used, let's update it.
white space and comments
Fix the stale filehandle issue with processing vuxml errors. It seems we are creating a new $fh value each time through the loop. While that approach worked with HANDLE, it does not work with IO::File. We were getting errors such as: no element found at line 1, column 0, byte 0 at /usr/local/lib/perl5/site_perl/mach/5.24/XML/Parser.pm line 187. I think that is indictative of an empty file. I encountered similar messages when printing the contents of the $fh before invoking the parser but those messages went away if I did a seek to te beginning of the file before parsing. I also noticed that the processing worked for the first vuln it processed, but never for the second. Invoking the script again allowed another vuln to get processed, and so on. That is what led me to the 'reuse' issue. I also tried a $p->DESTROY(); ... this also allowed things to work but this commit is a much better solution.
Remove spurious entries Adjust whitespace
Require SSL on connection
I think I've fixed the vuln issue
Better SQL
Updates not previously saved
fix trailing newline
Fix trailing newline
Fix trailing newlines
Add trailing newline
Better formatting
When adding a dependency to the database, strip any @py27 or :configure type stuff first.
Umm, move the code into the if, so it's not executed every time.
Remove generate_plist from updating the ports tree. That column has been removed.
Ignore empty lines
When there is no generate_plist, be sure to delete what's there.
Added when we have a bunch of FreshPorts::Ports_generate_plist=HASH(0x806d9f0d8) see https://svn.int.unixathome.org/viewvc.cgi/freshports-1/scripts/trunk/port.pm?view=annotate#l1234 I changed $this->{generate_plist} to $generate_plist
Add trailing '
Fix SPOOLINGDIR
We use constants. Declare that.
Remove unused code: * summary-all-days-create.pl * test_DailySummary.pl * test_FileBasename.pl
Remove log-munger code long not used
Remove old unused code: log-catcher
Adjust this to suit new directory locations
Deprecated in favour of queue-status.pl, used by Nagios.
adjust references to dynamic so they refer to the new structure
Fix the signals directory
Comment update
Avoid counting . files, such as .svn, which will artifically inflact the number
Use the correct user.
Add in missing my
Connect read-only
Set ConnectionType if not supplied.
Whitespace only
Connect as RO
Allow for no incoming ConnectionType and default it to commits
Remove FREEBSDCVSUPHOST, no longer user set DBUSER='reading', because most .sh scripts should be readonly
Add whitespace
When connecting, ask for a RO connection
* Add optional parameter when invoking FreshPorts::Database::GetDBHandle * use $FreshPorts::Constants::DB_ConnectionType => SOMEVALUE * possible values are: * $FreshPorts::Constants::DB_ConnectionType_ReadOnly * $FreshPorts::Constants::DB_ConnectionType_Commits * $FreshPorts::Constants::DB_ConnectionType_Listener
Add FreshPorts::Constants::DB_ConnectionType for use when calling FreshPorts::Database::GetDBHandle
* add user_listening and password * add user_readonly and password
Add three database connection types: * Commits - used by default * read only - will become the default soon * listening - used by fp-daemon
Sort by count and name
use MSGDIR for archiving, not something we make up locally
Remove blank line
Stop using HANDLE and start using File::IO This will hopefully avoid: Name "main::HANDLE" used only once: possible typo at ./process_vuxml.pl line 89. This should have been included in revision 4905
Stop using HANDLE and start using File::IO This will hopefully avoid: Name "main::HANDLE" used only once: possible typo at ./process_vuxml.pl line 89.
When not refreshing explain: this port *** EITHER *** does not need a refresh *** OR *** we were told not to fetch
Cater for different queues used on different hosts
We're going to a single tier queue. No more multiple queues.
Do not invoke LOGGER before it is defined via config.sh Add more logging calls
Alter this script so that each port is always updated. Remove ValueHasChanged() Become branch aware, because we have active ports on branches.
Make this script work
remove pkg_plist and makefile re https://github.com/FreshPorts/freshports/issues/53
Whitespace only
Add make-generate-plist.sh
Sometimes, you need to refresh from head..
Add more logging during initial processing of message. * Branch * Subject * message-id
Reinstate the depends_* trimming erroneously removed in a recent commit. While here, remove unnecessary references to SVNDIR_CHROOT
Add missing JailConfigurePlist
Add missing file
Use the correct path for perl
better logging
Add generate_plist feature. Fixes https://github.com/FreshPorts/freshports/issues/50 This will give us a better pkg-plist solution. Stop using full path names in depends field: * depends_build * depends_run * depends_lib * fetch_depends * extract_depends * patch_depends We were storing taking a As a result, we no longer need these functions: * freshports_ConvertPortPathToStandardLocation() * depends_stripper()
Use FreshPorts::Utilities::Report not syslog
Use FreshPorts::Utilities::Report
Start using FreshPorts::Utilities::Report
Add Report($;$) which takes a level and a message. Use this function internally. Start logging via local3 so we can capture that via syslog
Stop using . for concatenation
Invoke logger via local3.info to help direct all syslog messages to /var/log/freshports/
Rename to be like all the others
Adjust UMASK to be group writable (002) Adjust regex to consider only headers (:h) When discarding a message, log the Date, From, and Subject, much like a delivery.
Rename for better FreeBSD compatibility
We have a sample file, we should use it
A little more logging
Change /usr/bin/perl to /usr/local/bin/perl
Migrate to new directory layout
Document that it needs fixing with respect to directories
Add more logging. Make better use of to instead of cc
Specify Maildir despite using neither Maildir nor mbox. Why? I suspect it might make a difference in how the files are saved. Maildir saves to a temp file, then moves it o the final destination. We want that behavior. Under procmail, it was designed to save to a temp file, then mv. Now that we are using maildrop and Maildir, let's use their save/mv instead of doing it ourselves. Mind you, perhaps procmail did that too? Forwarding email address must be prefixed with a ! Stop doing the save/mv we did in procmail. Let Maildir handle that.
Document the move to maildrop
Add support for maildrop
Update, becase we don't do anything here, it all comes from the port.
Bring up to date with new directories. We no longer need foo+bar addresses because the recipient is now a dedicated user (i.e. not dan).
Add base to the list of part we process, despite it not being used yet. using https not http
Start using https instead of http
Adjust to the new directory layout to be introduced at 8xdtu.freshports.org
Remove, no longer required
I am not sure why we needed this directory
Use a constant for the email address
Reverse previous commit and leave config.sh in this directory. Instead, we will create a symlink to /usr/local/etc/freshports/config.sh Why? Pulling in a config.pm file for perl might be more difficult. Might as well use symlinks for both.
Move config.sh to /usr/local/etc/config.sh
Avoid new stuff on branches fixes #29
Add stuff for ports dri
Add a commit
For making new branches. This script should be run this by hand.
Add pkg-plist and Makefile to the things stored in the ports table
This refreshes all ports, not just those on head
Not having a distfile is not an error
Cater for new columns associated with no-packages https://github.com/FreshPorts/freshports/issues/9 - Not all ports have packages
Most of these changes allow for commits to ports on branches and the associated updates to any slave ports. * Add distinfo field to ports table * rename _GetPKGMESSAGE() to _GetFileContentsFromJail() because we are using it for both pkg-message and distinfo. * introduce CreatePortOnBranch()
pg_lasterror -> pg_last_error
Better debugging message whitespace
Better sleep and associated retry
The test must always be relative to PORTSDIR
* Master ports are always defined on head, because they may not exist on the branch It may occur that someone changes the master port on a branch, but highly unlikely * Remove extraneous fields from SQL query * pg_lasterror -> pg_last_error
Better retry / & delay times
pg_lasterror -> pg_last_error
add some debugging capability
Set distinfo.
Do the commit each time. Don't check distinfo.
Set master sites
make master-site-all has been replaced by make -V _MASTER_SITES_ALL
set $result before you test it. I have no idea why this bug did not arise earlier. That's terrible.
Keep track of elapsed time for svn up
Add pkgmessage to the ports table
Consider the interval, number of commits in past 5 minutes when reporting queue status.
set branch and add more debugging statements
Give a full path to HEAD
Properly quote the incoming file. re 201601062012.u06KC53V085831@repo.freebsd.org
use /usr/local/bin/perl
Set branch before proceeding
Add more checks and use the right dir.
Add comment regarding use of branches.
Give better starting points
whitespace
Whitespace only
Better debugging
Attempt to prepend $PORTSDIR to each depends
add
whitespace only
Use a newly created constant
Use the configuration user, not hardcoded user
Print my SQL
Move the flag removal to the end of the tasks
We have pkg version and pkg_version is gone.
Cater for branches, and set a starting point; you can change that later via code
Cater for branches
Allow for --fetch for fetching before running (defaults to 1) --message_id=MESSAGE_ID_FROM_COMMIT --port=www/bacula-sysutils
Allow one port to be refreshed
Correctly name some depends fields
Change the wrapper used because the old one has trouble with more recent perl? "This shouldn't happen at /usr/local/lib/perl5/5.18/Text/Wrap.pm line 84." -use Text::Wrap; +use Text::Wrapper;
Encode stuff
Better logging
Add error reporting
This servers is on FreeBSD 10 now
Better
Better logging
Remove AlreadyInserted code
Remove WHERE
This works.
Commits
require, not requires
This needs branches
$uses not uses$
Validate dates
Add FETCH_DEPENDS, EXTRACT_DEPENDS, PATCH_DEPENDS, and USES
Correct a print
Make it so we do not have to maintain this. Courtesy of bdrewrey
Update to a better formula
Add files we need in the jail
Cater for a couple more mailing lists
Now that host and dbname are in config.sh, remove them from here.
Add db connection information
Use ports_active, not ports_all, so we avoid deleted ports
updates
Use Mozilla::CA to deal with certificates
Add the changes to deal with branches
Works
Add. Incomplete.
Better error handling if pkg-descr goes wrong
Better debugging and make use of the jail paths
Fix Revision label on logging output.
Avoid duplicate entries in commit_log_ports_elements table
Commit it all!
Remove LooksLikeAMakefile(). We no longer get files via cvsweb, we get them via svn up.
Add savePortTableOnly()
Don't save on errors remove unused code whitespace
We might want to do a different kind of save
Cater for branch RELEASE*
Remove unused code Mention branch on error messages
Cater for yet another special Subject
Use eq not == for strings
Do not report failed dependency additions on branches
Cater for Subjects such as: Subject: svn commit: r341906 - in head: . dns dns/bind96 multimedia - 201401310024.s0V0OxDq013147@svn.freebsd.org - http://docs.freebsd.org/cgi/mid.cgi?201401310024.s0V0OxDq013147
Pass CommitBranch when saving port details
Cater for change to head proper, such as - http://svnweb.freebsd.org/base?view=revision&revision=261303 - http://docs.freebsd.org/cgi/mid.cgi?201401302226.s0UMQpDr065857
whitespace
Add SVNBaseDir
Not used
Update for Branches
Ignore everything after: Changes in other areas also in this revision
Correct the branch for the docs list
Now that we are using a perl module to get the List Id, fix the ID for the src list
use head, not HEAD.
Remove _FetchFilesNeedingRefresh_DELETE_ME()
Start using a better way of getting mailing list properties.
Correctly print the jails
Go some way to detecting commits to branches and split the filename accordingly
Remove Pathname_ID. It is not used.
include FRESHPORTS_FREEBSD_SVN_URL and PORTS_SVN
PORTSDIR is needed by process_vuxml.sh
PostgreSQL now has autovacuum
Get rid of this MySQL antiquity
cater for scheidell (ports committer)
Once you see Directory Properties, we're done.
Get the messages subject properly
Set long description / homepage
Remove PORTSDIR
comments out OPTIONSFILE
Add comment
Say which flag is not set
Don't test for an existing file from outside the jail. Drop the test entirely and just go get the contents.
Monitor incoming as well
no more Cwd or File-PathConvert.
add JailMasterSitesScript, JailShowConfigScript, JailDescr add _GetRealPath so we get the right description file
Start using a hostname for the database connection
add hostname to database connection
Add FreeBSDVersion = 9 I should find a way to do this without having the bump the versin all the time...
Add some more jail scripts
Fix comment
Add vars/sh
We do the installing from the configuration directory
Avoid an uninitialized value message. See http://news.freshports.org/2013/07/09/xmlwriter-upgrade-causes-fatal-error/
Use the jail to get the pkg-descr
To refresh certain broken pkg-descr commits
not sure why this wasn't already in here...
Remove commented out code
Fix the setting of port.last_commit_id on new ports
Add more jail stuff
use new jail for make master sites
Get the port name in there, and order the output
Fix how we update in place. Do this by deleting, then inserting. It gives better results without getting very complex on updates.
Update instructions for running
Add
Retire cvsup-local-tree
Retiring this script now that SVN is used
Add
Add samples for config.pn and config.sh
Also display the required sudoers entries
Add trailing ;
Update to use the jail scripts
Add
Fix up
Fix...
Add support for the jail
Add the scripts for creating and running the jail infrastructure.
Fix /UPGRADING and /MOVED
Update to reflect locations in SVN
Start using Text::Unidecode from converters/p5-Text-Unidecode to handle UTF-8 to ASCII conversion.
Better handle file translating
remove code not required in prod
add p5-Email-Mime
Update to use MIME ports
Update our list of things we need.
Do not syslog messages about the queues. We cover that now via Nagios
Add / to the db root prefixes
Update to full pathes
spell it right
match the results of _FetchFilesNeedingRefresh
Correctly name all columns in the result set
Add in the repo names
- update for storing svn revision in the commit_log table - start doing an svn up on the revision name, not the individual files
Tell us the failure value
- remove the unused function GetFreeBSDRepository(), which always returned src. - add GetRevision() so we know the repository revision for this commit
Updates, base on what I had in config.pm
- start using PORTSVN as the PORTSDIR for our local ports tree - this lets us use svn up to grab updates - add a sanity test for CATEGORIES to ensure it contains the primary - category - ^ that is useful for repo copies - UsualPortsTreeLocation is /usr/ports, not /usr - This is because our incoming file names are prefixed with ports/ to - reflect the existing tree structure in our database which was set up when FreeBSD used cvs for the ports tree, not svn as it - now does - start fetching files from svn using svn up - this will be useful for directories involved in repo copies. No more - missing Makefiles on svn mv. - in such cases, the file being fetched will be blank, but the directory is - not.
Do not do a -w on the perl call
Add another debugging statement Be more explicit on a regex: use .* instead of *
Trim any leading/trailing whitespace in file names
- instead of processing the entire vuln.xlm file, look at each one and compare the checksum. Process only the different ones
- Start using svn cat for fetching files.
- Append to the vuxml.log, don't just overwrite it - inlude some timestamps
Adjust for changes to vuxml_mark_commits... supply a named paramter
- start saving checksum - provide a method for fetching a checksum, based on vid
Adjust for the different format of the SVN tag
- adjust for new approach of updating just the vuln that have changed
- adjust for new approach of updating just the vuln that have changed - this script now just processed one vuln at a time - record the checksum - keep track of whether or not we updated the database so our caller can take action based upon that - add more detailed debugging statements
Do not fetch directories. Only files.
Remove ======
Handle the special case of 'ports' directory.
Assume ports repository and adjust the URL so it works...
Assume ports repository and adjust the URL so it works...
Ignore CVSROOT so we don't create a category and fetch it...
Start fetching from SVN, not CVS
Add the script for fetching files from SVN
Remove trail PROPS message Use the correct List-Id for the post-svn-head list
When element_id is null, use null, not a space
Now we know the List Id...
Rename a variable
- prepare for FreeBSD move of post to svn - alter svn processing so it prepends a repository name to all filenames - create process_mail.pm (FreshPorts::ProcessMail) which contains a few - common functions for parsing svn messages - move some functions from process_mail.pl to process_mail.pm - alter process_mail.pl to reference FreshPorts::ProcessMail - GetOS_RepoPrefix() is the function that gets the repo name - while here, remove process_cvs_mail.pl & process_svn_mail.pl which were deprecated by respective .pm files some time ago
If a given field is undefined, use '' instead.
Use the new FreshPorts::CommitterOptIn::GetErrorCount() function
Better comment regarding the main loop
Enhance an existing debugging message
Add GetErrorCount()
Correct a flag mentioned in usage statement
- better format a call to FreshPorts::CommitterOptIn::RecordErrorDetails() - add missing DEPENDS as a sanity test failure - add more debugging related to depends evaluation - in sanity test failures, mention DEPENDS that cannot be translated to a - dependency
Whitespace adjustment
Check for OFFLINE
Correctly spell the table name.
Fix, thanks to crees
Convert /./ to / in pathnames
More debugging output
Do for _RecordPortFilesTouchedByThatCommit() what we just did for _CompileListOfPorts()
Avoid a situation where a file at the category level is mistaken for a port. See 201205251025.q4PAPOvV092118@repoman.freebsd.org
Add more debugging information. This stemed from 201205251025.q4PAPOvV092118@repoman.freebsd.org
Allow for svn doc mailing list.
Add script for checking number of ports without entries in the categories table.
More comments regarding partial vuln processing
Mention where we'd need to change stuff for partial vuln.xml processing.
whitespace
Add some comments regarding partial processing of vuln.xml based on keeping an MD5 of each vuln
Remove pkg from the list of constants. Issue raised by bapt re ports-mgmt/pkg
Don't go deep
Remember to clear the ports_vulnerable table when processing
Do not die if no files are found. That can happen, say with an svn cp
If no valid expiration date, ignore the supplied value.
Change tab to spaces Put the / in 'quotes' to avoid division.
Add ip_address
Add another list
Add **OBSOLETE** CVS commit messages for the entire tree
Add missing pm
Add in projects
Some updates for port dependencies.
Introduce, and start using, process_svn_mail.pm for processing SVN commits. This has been in use on svn.freshports.org for quite some time. This change allows freebsd-cvs.sh to invoke process_mail.pl, which decides if this is an email we can process.
Set the value of FetchOK
Update
Have a max depth of 1.
Be sure to strip the depends before inserting.
Add
Add port_dependencies
Correctly spell license, and correct a bug which prevented license from being correctly picked up. REPORTED BY: Thierry Thomas" <thierry@pompo.net>
Add for Nagios
Add password token purging
The ability to delete old password tokens
Add.
Add support for license
Add another test.
Cater for empty change values
Add
Allow FreeBSD 8
Remove X11BASE
Set RESTRICTED correctly.
Keep the constants and utilities in there.
Commits to vendor branches slightly differs from normal commits. From Oleksandr Tymoshenko <gonzo@bluezbox.com>
Add new script amended by Oleksandr Tymoshenko (gonzo@)
Add better handling of cache update for things such as GIDs, UIDs, LEGAL, etc.
Now that we are not using ConvertFilePath, we aren't picking up the Freeze. Fix that.
Cast greatest() to ::date
We don't need to use db_escape() any more. DBI does proper PostgreSQL quoting.
Add details for FreeBSD 7. Incorporate changes suggested by koitsu@
Invoke NullIFEmpty on several more port fields.
Treat restricted and no_cdrom with special care for NULL
Correctly handle is_interactive to avoid setting that field to ENULL.
Start using NULLIfEmpty()
Remove _NULLIfEmpty() as it is now part of utilities.pm Start using the new location.
Move NULLIfEmpty() here from port.pm
Start escaping stuff going into the port table.
Start escaping the additions to the commit_log table.
Add a few debugging hints
Add db_escape, which escpaes stuff for PostgreSQL, E'stuff'
For showconfig: - use OPTIONSFILE=/nonexistent - remove LOCALBASE and X11BASE
Add ports.showconfig
Ready to go
Get the pathname right.
Wrong website. :)
Add, incomplete.
ignore OFFLINE
Don't echo the parm. check for valid parm, and if none, echo usage.
Check for OFFLINE
Add support for fp-listen
Add a log message
Remove irrelevant comments Add a full path to the OFFLINE file
Check for OFFLINE
Add a script for taking the website offline for maintenance
I now think ConvertFilePath is wrong. I should not munge the file name like that. Do it at fetch time!
Better handle failed master-sites-all
Cater for no vulns
Give a proper count of vulns and ports.
Invoke FreshPorts::Utilities::InitSyslog();
Do a better log message
Try syslog
Set the Queue as a configuration item.
We no longer patch
Use the native ports infrastructure instead of own patch MASTERPORT => MASTER_PORT
We no longer patch. The previous patch was to just keep the code instead of abandoning it.
How to patch on the fly.
we no longer patch
We don't patch any more
add ports/Mk/bsd.port.mk
We patch less for a while
No longer needed.
Remove. Covered by other patch: bsd.port.mk.master-slave-patch
Use more constants Fetch files before invoking scripts
Remove inaccurate comments referring to ./process_moved.sh Log more information This script no long fetches the file. That should occur before this script is invoked.
Log more information Use the CVSWEB URL from config.sh
When we can't fetch all the files, report an error and exit the loop.
Add a ConvertFilePath($FilePath) Cater for older commits, without a repository set in the message.
Remove PORTDIR from MASTERPORT. Some ports do this in a wonky way. For example: category | name | master_port --------------+----------------------------------------+----------------------------------------------- graphics | ocaml-lablgl | /usr/home/dan/ports/devel/ocaml-camlidl archivers | ocaml-zip | /usr/home/dan/ports/devel/ocaml-camlidl security | ocaml-cryptgps | /usr/home/dan/ports/devel/ocaml-camlidl x11-toolkits | ocaml-lablgtk | //usr/home/dan/ports/devel/ocaml-camlidl japanese | kinput2-canna+sj3 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+sj3+wnn6 | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+sj3+wnn7 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+wnn7 | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+wnn6 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-sj3+wnn7 | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-wnn7 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-freewnn+sj3 | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-wnn6 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+freewnn | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-canna+freewnn+sj3 | //usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-sj3+wnn6 | /usr/home/dan/ports/japanese/kinput2-freewnn/ japanese | kinput2-sj3 | //usr/home/dan/ports/japanese/kinput2-freewnn/ math | spooles-mpich | /usr/home/dan/ports/math/spooles/ science | mpqc-mpich | //usr/home/dan/ports/science/mpqc/ x11-toolkits | fltk-threads | /usr/home/dan/ports/x11-toolkits/fltk/
Add:
43 18 * * * ${DIR}/cvsup-local-tree
which replaces the old command line version... which started to get ugly
Add
Add FREEBSDCVSUPHOST
One of these compared items may be empty. So add an X so they aren't.
remove inaccurate comment
that should be sysutils/bacula-server, not bacula-server.
Add the master port tests
Test via the ports tree
patch with -N
Check that bacula-client is a slave port of bacula-server. We could make this a list of such ports, but that's for future work.
Comment out a debug statement.
Clarify the diagnostic message when a category is not found
Use FreshPorts::Config::Fetch_Retry_Limit and $FreshPorts::Config::Fetch_Sleep_Time
Add a \n
Add the names of the files which receive special processing.
Remove extraneous line
Take a better approach when checking for non ASCII files... Don't look for HTML anywhere in the file output. Just look for it at the start of the line.
Do not munge the result value when fetching a file
Provide knobs to set fetch sleep intervals and fetch retries. This should improve handling of network comms problems.
Remove 2nd instance of my declaration.
Better error logging
Add scripts-log
Let's do logger, not echo for these errors.
Give a better cvsup example
This should be run whether or not we are OFFLINE Simplify the script.
Add
Add these patches for the ports Mk system
Oh, we're using the old style of cache file name when flushing the cache. And we aren't finding anything. So the cache is never cleared. This is a problem.
process_moved.sh, process_updating.sh, and process_vuxml.sh are no longer wrong from crontab.
Start using a DIRLOG for keeping track of what happens
Add code to append the repo name to CVSROOT in each commit. Each cvs-all email contains a line of this form: FreeBSD doc repository where doc is the name of the repository. It is important to append the repository name to CVSROOT for any commits to the CVSROOT/ directory.
Use the new ReportErrorEmailNoPrint().
add ReportErrorEmailNoPrint() and alter _ReportErrorHelper() to conditionally print the error message.
White space only
Fix the ports freeze processing. It seems I broke with a previous commit. Reported by: Benedict Reuschling
Move back to the main cvsweb site.
Remove the hardcoded links to the design contest.
AnnouncementsGet() -> AnnouncementsGetPlain()
Add http://www.freshports.org/DesignContestVoting/
Start using FRESHPORTS_FREEBSD_CVS_URL While here, output more useful debugging information.
Fixed typo
add FRESHPORTS_FREEBSD_CVS_URL
Add more debugging output
add ~checkout~ to the URL we are fetching. This avoid getting HTML in cvsweb3.
./perl -> perl
The most-watched.pl needs to have a path to dynamic, and trailing ; before the }
Add
Add count of file in the ports tree
Oh, if you find a category that does not exist, that is OK. It just has not been populated yet. No worries.
Add a trail \n to debugging. Mention the file we are creating for www/en/ports/categories processing
Add textproc/p5-Text-CSV_XS as a dependency
Add special processing of www/en/ports/categories
Add a script for updating the category description based upon www/en/ports/categories
Order them right....
Order by date, lower(name)
Toggle sense of new for a vuln
Put the * for a modified vuln outside the hyperlink
Put t/f in output for a modified vuln
Remove line left over from merge.
Merge from FreshPorts2 to HEAD
Update copyright
send one message per user
move RefreshMainPage and GetMaxCommitLogPortId to cache.pm
if commiting a port, call DailySummaryDateAdd
also check the daily_refreshes file
reflect new location of CreateDailySummary in cache.pm
move CreateDailySummary to new cache.pm module
No longer needed
Move freshports_ConvertPortPathToStandardLocation to port.pm
spelling mistake changes, etc.
print the file name on a file action error
Be a little more specific about the start of the files section
create msgs directory
scripts for comparing the database with a list of category/port entries.
remember to update the categories field. pr 114
PR 112 watch_notice_log.frequency_id needs to be implemented changing frequency to frequency_id check scripts/email.pl and AddToLogs
RecordLastestPortCommits now works independently of date.
update copyright
call freshports_ConverPortPathToStandardLocation
There is no need to grab the commit_log id first. It will be obtained by the commit_log class.
Add freshports_ConverPortPathToStandardLocation, which will change path names to the standard /usr/ports pathname
Add Makefile.inc to the list of things we ignore
Let's try distinct, just in case...
Put the committer into the output
Mark those ports found in the INDEX and those not found.
*** empty log message ***
sleep at the start of the script, not the end. if an error occurs, we don't retry it right away.
update copyright
update copyright change return value of 3 to 0 if all is well we don't move files around. the fp-daemon script does
add retry directory and update copyright
need to create the directories
we no longer need lynx
display the full file name when processing
Check for blank branch name
All make commands must refer to the correct ports tree via PORTSDIR=
IF YOU CHANGE THE MAKE COMMAND, CHANTGE THE SPLIT!!!!!!!!!!!!!!! we'll have to refresh all ports now.
adjust debugging code
let's add a commit-rollback just for fun.
Don't die with an error, just report it. We will clear refresh_now so the SP doesn't have to.
we need the committer in here.
let's use fetch instead of lynx
change LooksLikeAMakefile to be correct more often
remember to install procmail
add .forward
missed this
Add these into the repo
let's make this into a daemon-type script for use with daemontools.
at the end move the input file away to the raw directory
remember to include housekeeping module
New class.
Start using the housekeeping class
Don't commit here when refreshing ports. We'll do that lower down in the code.
After refreshing a port, update housekeeping.
Slight adjustment to error returns
Let's have a look at the Makefiles before we use them. If they are ASCII Text, then they are probably a Makefile. If not, they are probably HTML, and that means the fetch failed.
only refresh non-deleted ports
only do the table update if a new port has been committed.
no longer any need to create daily summaries here.
no longer a need to touch a file after refreshing.
how about only doing the SQL if there is a need to do so
To speed up the home page, we are going to cache the data in a table. The main page will then read from that table. The main page will be updated every minute.
whitespace changes only
Start using "make master-sites-all" instead of "make -V MASTERSITES"
Refresh all ports.
refresh-unrefreshed-ports.pl stays as it was, refreshing unrefreshed ports based on commit_log_ports.needs_refresh.
Add a flag which allows us to refresh all ports. useful during fp1-fp2 migration
script to create ~/msgs
Add flag -R which means do not refresh any ports associated with a given commit. Useful during the fp1-fp2 migration. We'll just refresh each port after the data is imported.
If an invalid file action is found, display it as well as report it.
don't export deleted ports - they won't exist in the ports tree and we won't be able to refresh the port. This does mean we lose the history for about 451 deleted ports. I can live that that. We can always import them later.
allow incoming parameter to set limit If file_action = 'I' (import) used add and change filename to '.../Makefile'
Add incoming limit parameter
remember to clear out MessageTo in case there is more than one recipient.
don't fetch stuff if -r flag is set.
oops, use PST not EST
Here's the file where will allow us to export FP1 data into XML format. Looks OK so far. But will need further testing of output
Create a monthly compression archive. note that the other compress script appears to be unused.
make sure utilities is included where it should be included. and that config is included in utilities.
if the code dies, email me
Use new error routine
Add new commit_log class Start using new ReportError function from utilities.pm instead of calling SysLog and die.
make sure to use " not ' when expanding $variables.
doh! do not hard code path names!
I should test more before committing.
When an error occurs when processing a make file, return the error to the calling function. Then, don't update the port because it's not ready
Put a knob in to turn off creation of a pkg directory. I think I added this to detect problems which arose if this directory exists. noteably: pre-everything having two : after it.
umm, ports-supfile is in ~/scripts not ~/
We need a custom supfile for this. Let's also add a cvsup to the crontab.
PR#21 cater for imported sources. With a recent change implemented by Josef Karthauser, imported source cvs-all message do contain a BRANCH. However, the message do not contain a LOG section. Thus the XML field is empty. Get around the problem with a minor reorganization of the code. Grab the message subject. Grab the log. If the log is empty, use the subject. ALSO: white space changes.
Add copyrights
add copyright
Add copyright
convert to PostgreSQL
Add copyright
Start using ${HOME}
upgrade to postgresql. Now that this is done, the next step is to get some of you on the Develop list to start tracking your ports via fp2.freshports.org. BUT FIRST I'll have to upgrade that site to the latest database and scripts as found in lists-test.freshports.org. Watch this space.
Add in the copyright notice. fix up the paths to refer to HOME
add another simple test
remove debugging code
simple whitespace change
$hash{portname} is not the same as
$hash{$portname}
Fix this annoying bug so more than just the first port in a commit gets refreshed.
daily summary changes - use category/port format - add time-ajustment factor to cater for time zone differences
convert to postgresql
- correct where the script was still using ports.needs_refresh instead of commit_log_ports.needs_refresh. - for now, we will fetch every file touch by a port commit. I still don't have a solution for files included by Makefile.
Use $ENV{HOME} not ~/
Handle deleted ports better. Remove the path-to-ports so it works with symlinks.
Add some real-path calls so we have the real pathname instead of symlinks.
oops. fix a typo
slight whitespace change to make SQL easier to read
after refreshing a port from newly fetched files, let's actually save the results to the database.
in CreateDailySummary, use commit_log_ports.{port_version|port_revision}
handle port_version
move the commit to be before the CreateDailySummary call. If that call falls over, we want to be sure the transaction is closed. otherwise, it will roll back and undo what we did.
update to use commit_log_ports.needs_refresh not ports.needs_refresh
- with master/slave ports, the directory we are fetching to (master port) may not be the directory we are working in (slave port; the one being refreshed. adjust fetching scheme accordingly. - add function to remove port directory prefix (RemovePortsPrefix).
start using $ENV{HOME} so we know the correct home directory without modifying
this config file.
slightly modify the SQL to match similar query in refresh-unrefreshed-ports.pl
Update this script to use commit_log_ports.needs_refresh instead of ports.needs_refresh PR56
use relative paths, not absolute
Add call to script to archive out messages.
change default pathnames to be relative not absolute
It appears we have a working solution for storing port version with each commit.
My first attempt at moving needs_refresh from the port table to the commit_log_ports table. In addition, the commit_log_ports will hold the port_version associated with that commit. Thus, as you look down the page, you should be able to see the port_version change from commit to commit (not that every commit causes a change).
remove comments no longer relevant
No longer needed here
Add the new archive-messages.sh. call this from cron. it will call the improved archive-logs.sh which will move the msgs away from the main directory and archive them.
correctly detect the successful opening of a file for our daily summary output.
test for packages.exists file before we grep from it
Upgrade to FP2
provide a new option: -O (Overwrite) when processing an XML file, if the supplied message id exists, delete it and continue processing. Useful when you need to reprocess a message for some reason.
use the commit_log_ports table, not commit_log_port.
upgrade to FP2
put default values which make sense but can't be used.
Remove some commented out code
First commit
Add copyright notices and Id tags to each file.
Add copyright notice expand messages when Makefile errors occur
Add copyright notice start using the new and improved commit_log_ports
add some testing scripts
first commit. includes instructions about installing perl modules for these scripts.
don't save commit_log_id unless it is supplied.
Add some basic code to cater for master/slave ports.
ensure we take the commit date before we clear out all the date values.
add missing ) to an if. actually write HTML to FILE.
If the version isn't set, don't try to display it
Don't do the build of daily summaries until after we have refreshed the ports. Otherwise for new ports, version will be empty.
Don't do any of the port update things if there are no ports. This mainly reduces the size of the log file.
don't quote the date
Remove trailing / from 'use lib' statement.
Add CreateDailySummary to the list of things that happen when a message is processed.
If a port has been deleted, make sure we delete the element which corresponds to that port.
Move things around so they work better.
remove CommitLogElementsInsert replace EnsureCategoryAndPortExist with SaveChangesToPortsTree create _CompileListOfPorts which replaces the fuctionality of EnsureCategoryAndPortExist Instead of saving a port as we go along, we will compile a list of ports based on the files we find. Once all files are processed and saved to the database, we have our list. Then we process the files again, setting the needs_refresh value for the port affected by this file. If we find that the Makefile for a port has been deleted, we set needs_refresh = 0.
use a hash instead of a character string for ignored files
first commit
Make sure we are looking for " Revision" at the start of the line. See 200111231716.fANHGwN21197@freefall.freebsd.org for what broke this.
if the port is new and has not been deleted, then set needs_refresh.
remove unneeded exports
if the port has been deleted, make sure we set port->{deleted} correctly
and process things.
move the refresh ports process to port.pm so it can be used
both with load_xml_into_db.pl and from refresh-unrefreshed-ports.pl
separate out the refreshing of a port from the saving of a commit message.
Always check if the ElementRevisionExists before doing a ElementRevisionInsert.
Spell the path correctly
remove exta my.
use SystemID not SystemBranchID
log it when we create a new branch
fix typos
rename system_version to system_branch rename system_version_element to system_branch_element_revision
check for failed file open
clean up code, check for port and category before deciding this is a port
PR 4 solved. scan the lines following the subject and include them as part of the subject if they start with 9 spaces.
fix typo. should be OBTAINEDFROM not OBTAINED in the dtd.
The previous commit was in error. PR 15 Previously the log message was terminated at the first occurance of: Revision PR: Submitted by: Approved by: Reviewed by: To Unsubscribe Now it's only: Revision To Unsubscribe Also, populate the ObtainedFrom field while we are here.
PR 15 Previously the log message was terminated at the first occurance of:
spell receive
add the revision field.
Use commit_log_element
Use new class: commit_log_port
use ne instead of eq. condition was wrong.
updates
add missing files
Get those values from the result set.
save the right stuff when saving. it still needs to be read back in.
use the right id
Now we fetch the files needed to refresh the port, we update the port class, and we save it. port.pm needs to be updated to save and fetch more fields.
set the commit id before saving
Move ReadFile to a common location.
remove code we've already converted
correct the sql for updating a port
remove unneeded comment
when a port is created, needs_refresh is set correctly. now we need to set that when a port is updated.
first commit
needs_refresh is now being computed. We have have to do something special for ports which are created first time.
Now processing the files and ports correctly. now we just have to parse the file names and update the port needs_refresh.
We now have a list of ports which needs to be updated. next step: set the needs_refresh field and then save the results back to the database.
Create a hash of ports...
whitespace changes only
fetch by partial name
checkpoint
just a checkpoint
Ummm, let's get the element_id of the port before we create the port.
Remove GetDBHandle and use the one provided by FreshPorts::Database. use $dbh not $dbh_pg
die if "is_primary" is not set.
use the recently created FreshPorts::Category and FreshPorts::Port classes
Move GetDBHandle to database.php Add test_port_save
Just another checkpoint
Make save work.
first time
Add the ID to the file
simplify things greatly
backups are handled by another user now
try $HOME instead of /usr/local/etc/freshports
better the grammar
instead of port/category, try category/port
fix path to lastupdate.
provide details on each port to be refreshed
fix typo
update path
fix typo in last commit
fix path
Put the database connect into freshports_database.pm and use freshports_database.pm.sample as the example.
update
get the path right
get the directory correct
updates
tidy up
fix up crontabs
Send out individualized messages for the ports changes.
pick a better password
Add some cache busting code to the fetch.
fix syntax error
Don't create a pkg directory in fetch-cvs-file.sh. Let's do it where werun the make -V. then we'll remove it. this catches the pre-everything bug.
Don't create a pkg directory here. Let's do it where we run the make -V. then we'll remove it. this catches the pre-everything bug.
Don't chown any more
change path. This should really become a local configuration item
fix dir path
change paths
change path
Update locations
change locations
fresh updates from the website
first commit
first commits
update the scripts with the latest from the website
*** empty log message ***
First commit
use a LIB command.
Added logs of time stamp printing so I could see where things took up time.
CreateDailySummary mods to increase db performance
*** empty log message ***
Add code to strip timezone from timestamp. Might be related to new mysql. Modified Files: updates.pl
Hmmm, it assumed that a port commit started with cvs commit: ports/ now, it doesn't. the ports/ bit can be anywhere on the subject: line. Modified Files: log-catcher.awk
oops! forgot to add the ports changes for pkg-descr etc. Committing in updates Modified Files: updates.pl
changed pkg/DESCR to pkg-descr and other changes associated with new ports layout. Modified Files: ports.pm
extra checks to ensure directories exist. Modified Files: fetch-cvs-file.sh
first commit Added Files: fetch-cvs-file.sh
let the owner know if a category is created. Modified Files: updates.pl
Cater for ports which are deleted, then reimported. We have them in the tree, but they are deleted. Just set them "status = 'A'" and get on with it. Modified Files: updates.pl
add GetPortCategory add category to PortCreate add GetPortID Modified Files: updates.pl
damn, forgot the db login Modified Files: updates.pl
another path which needs changing. Modified Files: updates.pl
change db and password to values which obviously need changing. Modified Files: updates.pl
add fake paths so we don't updates stuff we shouldn't. Committing in . Modified Files: log-catcher.awk
final changes before freshports 1.1 Committing in . Modified Files: log-catcher.awk log-catcher.pl Modified Files: updates.pl
final changes before freshports 1.1 Committing in . Modified Files: log-catcher.awk log-catcher.pl
rename change_log_details.change_port_id to change_log_port_id Modified Files: updates.pl
move the call to MarkPortAsRefreshNeeded to where it is always called for every entry being modified. Failure to do this means pkg/DESCR are not fetched and therefore the data is out of date. Modified Files: updates.pl
add a zero to the mask (to make it octal? or hex?) Modified Files: portschange.pm
now tests for a failed make call. Modified Files: ports.pm
The big change to full automation of commits.
creates a daily summary for all change_logs in system.
initial commits.
now includes code so you can refresh a single port. Modified Files: fetch-refresh-ports.pl
Looks good so far. Just a quick commit to save everything. At this point, I have the fetching of Makefile, pkg/DESCR, and pkg/COMMENTS going. But I have to know when to fetch such things. Committing in . Modified Files: fetch-cvs-file.sh fetch-refresh-ports.pl ports.pm test.pl
fetch-cvs-file.pl is actually fetch-cvs-file.sh Committing in . Modified Files: announce.pl email.pl ports.pm refresh-ports.pl test.pl Removed Files: fetch-cvs-file.pl
These files are used to refresh a port on-the-fly. fetch-cvs-file.sh will fetch the latest version of any cvs file. fetch-refresh-ports.pl uses the above to refresh all ports which need refreshing. Added Files: fetch-refresh-ports.pl fetch-cvs-file.sh
latest and greatest Modified Files: log-catcher.pl log-munger.awk
fixed up setting of refresh_needed and started using index instead of a regex for FilesWhichPromptRefresh. Modified Files: updates.pl
now updates change_log_details properly. obtains needs_refresh and status when checking for existing ports. assigns value to needs_refresh only if necessary. sets needs_refresh to "N" if port is deleted. Modified Files: updates.pl
use index function instead of a regex when searching for existing pacakges. Modified Files: ports.pm
now records the changes in distinct tables as they arrive Modified Files: updates.pl
now handles an undefined $homepage much better. Modified Files: refresh-ports.pl ports.pm
updated the outgoing message Modified Files: ports.pm
Added SendWatchNotice Modified Files: ports.pm
now marks ports as deleted if the Makefile is removed.
no longer sets status to 'N' when a port is updated ignored ports are now tested for individually as using a REGEX was causing too many problems. improved formatting of code.
When a new port is added, the status is now set to 'A' - active.
now works fine.
split the file name into three parts, so we can recognize special files such as pkg/DESCR, etc.
All the scripts for log catching, log munging, and dataabse update.
finally created a perl module for the most common routines.
Standard project directories initialized by cvs2svn.