reorganize "files" (binaries) folder and add to repo

[IvanExpert]

Ok, need your help here, read the whole thing:

I'm in the process of reorganizing the A2SERVER files folder (which is mostly binaries) to have more consistent names, e.g. "foo-rpi.tgz" for Raspberry Pi, "foo-debian_x86.tgz" for Debian x86, "foo-debian_x64.tgz" for Debian x64, and "foo-debian7_x86.tgz" for something specific to Debian 7 (e.g. due to some dependency on an older package not supplied in Debian 8).

Since these binaries are as important a part of the install process as the scripts, I want to add them to the repo, excluding the enormous OS distribution files, which will get a separate folder (e.g. "rasppleii/dist"). And then I want these binaries to be supported by the A2SERVER_SCRIPT_URL export. So far, so good, I can do all that.

Here's a problem I don't know quite how to resolve: Several files, e.g. unar-rpi.tgz, nulib2-rpi.tgz, and raspbian-update.txt, are shared by A2SERVER and A2CLOUD. I don't want to potentially have redundant and potentially inconsistent versions in the A2SERVER and A2CLOUD hierarchies. In the bad old pre-Github days, I logged into the web host's shell and simply made symlinks to a single file as a way to avoid this.

However, that isn't gonna fly now. My inclination is to put common files in rasppleii/common/files and then change any links in the scripts to use that. I've already started to do this. However, this poses two problems:

• As the A2SERVER_SCRIPT_URL shell variable is currently implemented, it would require ugly reach-backwards URL's like

wget -qO /tmp/somefile.tgz ${A2SERVER_SCRIPT_URL}../rasppleii/common/files/somefile.tgz

• A developer can't clone just the rasppleii/a2server project -- they also have to get rasppleii/rasppleii

The first would seem to be addressable by a more generalized A2SERVER_SCRIPT_URL, e.g. RASPPLEII_SCRIPT_URL, and then the A2SERVER URLs would all need to specify "a2server/", e.g. ${RASPPLEII_SCRIPT_URL}a2server/etc/etc.txt

The second is would seem to be addressable by indicating rasppleii/rasppleii as a dependency which can be requested during the git clone, though I don't know how to do that; or just clear instructions that the developer needs both.

Or, maybe there's another way to maintain consistent binaries in separate hierarchies in Git; if so, I don't know how to do that either.

TL;dr: What do you propose we do to make the 'files' subdirectory part of the repo but also keep things consistent for common files?

(Of course, this also confuses issues somewhat because Raspple II is being used generically as the parent of A2SERVER and A2CLOUD, even for non-Raspberry Pi setups, but that's, I think, just how things are gonna be, e.g. the name of this whole repo.)

[knghtbrd]

This is a complex problem. First, binary files don't work well in VCS in general, since each revision of the file must be downloaded. Moreover, every time you clone a DVCS repository like github, you will download every single version of every single binary in version control. That means if you WERE trying to put the multi-gigabyte VM images in traditional git hosting, you could find yourself trying to download terabytes within a year of releases! There's a git largefiles extension for getting around this, but the most recommended thing to do is store your source code in VCS and compile the binaries with checkout hooks (including cross-compiling.)

But that's still a big problem, and one of several inter-related problems. The first is a relatively small bite, but it sets some groundwork in motion: The common scripts (and their dependencies). The scripts common to both A2CLOUD and A2SERVER appear to be simply raspbian-update and cppo. Let's deal with cppo first.

First, it doesn't make sense to have cppo by itself. There are other scripts that do go well along with it: acmd, dopo, mkpo, dos2pro, and shk2image seem like they go with it. Seems like the dependencies for those tools are AppleCommander and nulib2.

Is AppleCommander even being maintained anymore? It appears that it may not be anymore, but that might just be that not much has been needed since Google Code shut down. We've discussed that it's kinda weirdly split between Google Code and SourceForge before.

If it's kinda orphaned upstream, we might just fold it along with the scripts into a new repository, which is one of the things you're suggesting. That removes some duplication from both A2SERVER and A2CLOUD.

The other shared script is raspbian-update which I have always had some mixed feelings about. I'd suggest leaving it alone for the time being and just accept that there are two copies for now. I'd rather there be only one, and I'd rather it not be part of either A2CLOUD or A2SERVER in the end. But I'm much better able to sell myself on the idea of removing from both once a lot more stuff has been done toward doing things the Debian way.

Before I blather on about potential solutions to anything else, I'll stop here so that we can make sure we're in the same book, to say nothing else about on the same page. :) How's my thinking thus far?

[knghtbrd]

I'm building a full dependency list for A2SERVER and I'm a little confused by some of the logic used in picking packages for wheezy/jessie for netatalk:

        if [[ $isJessie ]]; then
            if [[ $isRpi ]]; then
                getOldPackage "http://mirrordirector.raspbian.org/raspbian/pool/main/d/db/libdb5.1_5.1.29-5_armhf.deb"
            elif [[ $isDebian_x86 ]]; then
                getOldPackage "http://ftp.debian.org/debian/pool/main/d/db/libdb5.1_5.1.29-5_i386.deb"
            fi
        elif [[ $(apt-cache search '^libdb4.8$') ]]; then
            sudo apt-get -y install libdb4.8
        elif [[ $(apt-cache search '^libdb5.1$') ]]; then
            sudo apt-get -y install libdb5.1
        else
            break
        fi

I get isJessie there—getting the packages from wheezy if they're not installed so that you can use the older netatalk 2.2.4 binaries without recompiling them. What's the libdb4.8 there for? And preferring it to libdb5.1?

[IvanExpert]

I get isJessie there—getting the packages from wheezy if they're not installed so that you can use the older netatalk 2.2.4 binaries without recompiling them.

That's actually not the reason; I already have Jessie Netatalk binaries for both arm and x86, compiled based on the newer packages, and this was my original approach. However, I instead opted to use the older packages because 2.2.4 is specifically qualified for use with them, and I was unsure whether libgcrypt20 and libdb5.3 are specifically supposed to be backwards compatible with older versions. With that said, I don't see any ill effects when compiled with the newer packages, and others have reported success (by default, it will fall back on source which has transitional packages available), so how I'm doing it for now could be revisited.
What's the libdb4.8 there for? And preferring it to libdb5.1?

This might no longer be valid. I think Netatalk 2.1.x, which is what I started with, may have not explicitly supported 5.1, and so my bias was towards 4.8, which I'd seen proven to work and was explicitly supported; and whatever version of Ubuntu (before I switched to Debian) I was using at the time may not have offered 5.1. However, Raspbian didn't offer 4.8, so I installed the later but less proven version if that was the only option. I tend to be very much of the if it ain't broke don't fix it school, and also the updates might or might not cause problems school. However, there's probably no reason to not always install 5.1 at this point if it's available.

[IvanExpert]

I'll prioritize lib5.1 over libdb4.8 for the next commit.

[knghtbrd]

Should I drop 4.8 then? I'll leave you to decide whether to use the
newer Jessie packages otherwise or not.

I've got a pending commit that collapses some apt-get -y install
lines for build-dependencies to one line and explicitly comments that
those are build dependencies. Makes a few lines a bit long, but will
help later on I think.

Joseph

On Sun, Dec 20, 2015 at 07:05:11PM -0800, IvanExpert wrote:

I get isJessie there—getting the packages from wheezy if they're not installed so that you can use the older netatalk 2.2.4 binaries without recompiling them.

That's actually not the reason; I already have Jessie Netatalk binaries for both arm and x86, compiled based on the newer packages, and this was my original approach. However, I instead opted to use the older packages because 2.2.4 is specifically qualified for use with them, and I was unsure whether libgcrypt20 and libdb5.3 are specifically supposed to be backwards compatible with older versions. With that said, I don't see any ill effects when compiled with the newer packages, and others have reported success (by default, it will fall back on source which has transitional packages available), so how I'm doing it for now could be revisited.
What's the libdb4.8 there for? And preferring it to libdb5.1?

This might no longer be valid. I think Netatalk 2.1.x, which is what I started with, may have not explicitly supported 5.1, and so my bias was towards 4.8, which I'd seen proven to work and was explicitly supported; and whatever version of Ubuntu (before I switched to Debian) I was using at the time may not have offered 5.1. However, Raspbian didn't offer 4.8, so I installed the later but less proven version if that was the only option. I tend to be very much of the if it ain't broke don't fix it school, and also the updates might or might not cause problems school. However, there's probably no reason to not always install 5.1 at this point if it's available.

---

Reply to this email directly or view it on GitHub:
#51 (comment)

[knghtbrd]

Likewise, drop libssl0.9.8?

[IvanExpert]

This is a complex problem. First, binary files don't work well in VCS in general

Duh, don't know why I didn't think about that. Of course not.

but the most recommended thing to do is store your source code in VCS and compile the binaries with checkout hooks (including cross-compiling.)

Eww. I'd rather have checkout hooks to just download the binaries from elsewhere. In nearly every case where there are binaries, the source code does exist somewhere already, and I don't want to replicate it; plus, compiling on every checkout is slow and ugly. In some cases, the binaries are actually things like Apple II floppy disk images, and while theoretically those could be built at checkout with things like AppleCommander, yuck. What perhaps I'll do is make a tarball of the binaries, host it on my web site, and in the developer README it can say where to grab and extract them for local or alternate-server installations with A2SERVER_SCRIPT_URL.

First, it doesn't make sense to have cppo by itself. There are other scripts that do go well along with it: acmd, dopo, mkpo, dos2pro, and shk2image seem like they go with it. Seems like the dependencies for those tools are AppleCommander and nulib2.

Au contraire. There's a good reason to have cppo by itself: a dependency for AppleCommander is Java, which is NOT something I want to require for an A2SERVER-only installation. So I'm just as happy leaving those non-cppo items in A2CLOUD only. (Though I suppose dopo could be included in A2SERVER, as it has no dependencies.)

The other shared script is raspbian-update which I have always had some mixed feelings about. I'd suggest leaving it alone for the time being and just accept that there are two copies for now.

I can live with that as a short term solution for cppo and rasppleii-update, though I dislike the idea of these scripts forking as a result of being in two different projects, which it seems like could easily happen. I'll try to take care to make sure that if I update one, I update the other, but that'll be a hard thing to rely on.

[IvanExpert]

Don't drop 4.8 entirely; someone might want to install on an older Linux that doesn't have 5.1. In my next push I'll have 5.1 and 1.0.0 only for runtime, and fall back on 4.8 for dev; and I'll roll a new binary for netatalk based on 5.1 and 1.0.0.

[IvanExpert]

I've got a pending commit that collapses some apt-get -y install
lines for build-dependencies to one line and explicitly comments that
those are build dependencies. Makes a few lines a bit long, but will
help later on I think.

I'm hoping I'm gonna be done with my end of things soon; I think my preference would be for you to hold off on pushing your refactoring until I get 1.3.0 out the door, after which you can rework everything as you see fit.

[knghtbrd]

No serious refactoring, just adding comments and collapsing two adjacent lines into one, like so:

diff --git a/scripts/a2server-2-tools.txt b/scripts/a2server-2-tools.txt
index b7142f8..e5f395d 100755
--- a/scripts/a2server-2-tools.txt
+++ b/scripts/a2server-2-tools.txt
@@ -40,9 +40,10 @@ if ! command -v nulib2 > /dev/null; then
             touch /tmp/a2server-packageReposUpdated
         fi

-        sudo apt-get -y install build-essential
-        sudo apt-get -y install zlib1g-dev
+        # Dependencies: build-dep for nulib
+        sudo apt-get -y install build-essential zlib1g-dev
         sudo apt-get -y clean
+
         cd /tmp
         rm -rf /tmp/nulib &> /dev/null
         mkdir /tmp/nulib
@@ -78,6 +79,7 @@ if ! command -v unar > /dev/null; then
         touch /tmp/a2server-packageReposUpdated
     fi

+    # jessie and later: Just use the unar package
     if [[ $isJessie ]]; then
         sudo apt-get install -y unar
         sudo apt-get clean
@@ -85,8 +87,11 @@ if ! command -v unar > /dev/null; then

     if ! command -v unar > /dev/null; then
         if [[ $isRpi || $isDebian ]]; then
+
+            # Dependencies: for unar
             sudo apt-get -y install libgnustep-base1.22
             sudo apt-get clean
+
             if [[ $isRpi ]]; then
                 wget -qO- "http://appleii.ivanx.com/a2server/files/unar-rpi_wheezy.tgz" | sudo tar Pzx
             elif [[ $isDebian ]]; then
@@ -96,10 +101,11 @@ if ! command -v unar > /dev/null; then

         # If all else fails, compile from source.
         if ! command -v unar >/dev/null; then
-            sudo apt-get -y install build-essential
-            sudo apt-get -y install libgnustep-base-dev libz-dev libbz2-dev
-            sudo apt-get -y install libssl-dev libicu-dev unzip
+
+            # Dependencies: build-deps for unar
+            sudo apt-get -y install build-essential libgnustep-base-dev libz-dev libbz2-dev libssl-dev libicu-dev unzip
             sudo apt-get clean
+
             rm -rf /tmp/unar &> /dev/null
             mkdir /tmp/unar
             cd /tmp/unar
@@ -121,7 +127,9 @@ fi

 if ! command -v unzip >/dev/null; then
     echo "A2SERVER: Installing unzip..."
-sudo apt-get -y install unzip
+
+    # Dependencies: unzip
+    sudo apt-get -y install unzip
 else
     echo "A2SERVER: unzip has already been installed."
 fi

The idea I had was to do that kind of thing for apt-get lines, then go back and do likewise for wget lines so it's very easy to search for the next dependency. It doesn't have to be done now though if you're in the midst of preparing a commit.

Actually in that file, you can see that unzip is conditionally installed to satisfy a build-dep, then installed unconditionally so it's available right afterward. I did plan, after this commit, to move the unzip install above the one for unar and remove unzip from the extra build-deps for compiling unar from source, since it's already a dependency of A2SERVER as a whole. Moving about four lines up a block—not much of a refactor—but I don't want to do anything significant to it until after 1.3.0 is ready.

[IvanExpert]

Ok, that all seems pretty reasonable. I expect 1.3.0 to be ready shortly.

[knghtbrd]

Referring to my thoughts about collapsing apt-get lines a little or a repo for acme and related scripts and all?

I'll comment about the next bit I've got in mind that all seems good.

Joseph
Sent via mobile

On Dec 20, 2015, at 21:28, IvanExpert notifications@github.com wrote:

Ok, that all seems pretty reasonable. I expect 1.3.0 to be ready shortly.

—
Reply to this email directly or view it on GitHub.

[IvanExpert]

Referring to collapsing apt-get lines, and specifically not referring to repo for acmd and related scripts, other than cppo (and maybe dopo). The scripts that depend on AppleCommander belong in A2CLOUD, because if they require AppleCommander, then they require Java, and A2SERVER should not require Java. (A2CLOUD by definition requires Java because the ADTPro server is in Java.)

[knghtbrd]

Saw your other replies later (iOS Mail…) I'll stick to the trivial
stuff for now and not move any files for the time being so as not to
slow down 1.3.0 development.

Joseph

On Mon, Dec 21, 2015 at 07:15:37AM -0800, IvanExpert wrote:

Referring to collapsing apt-get lines, and specifically not referring to repo for acmd and related scripts, other than cppo (and maybe dopo). The scripts that depend on AppleCommander belong in A2CLOUD, because if they require AppleCommander, then they require Java, and A2SERVER should not require Java.

---

Reply to this email directly or view it on GitHub:
#51 (comment)

[knghtbrd]

Just push that, had a minor merge conflict where you added an apt-get update where I added a comment. Correct solution was obviously to have both.