Discussion:
[sword-devel] Alternative method to mark translation changes
David Haslam
2010-11-04 22:55:27 UTC
Permalink
In the KJV module, there are 17 occurences of the alternative method to mark
translation changes.
<seg subType="x-added" type="x-transChange">...</seg>This alternative method
is a hack which seems to be required when the transChange text is within a
<w> container.
I assume that OSIS does not permit the recommended
<transChange>...</transChange> container element in such circumstances.

Xiphos 3.1.3 displays these without italics. This is a bug.

I have also observed that diatheke with RTF output_format does the same, so
perhaps the bug is in the SWORD API.

See also http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach
http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach

I have also reported the bug in the Xiphos tracker. ID=3103244.

David
--
View this message in context: http://sword-dev.350566.n4.nabble.com/Alternative-method-to-mark-translation-changes-tp3027931p3027931.html
Sent from the SWORD Dev mailing list archive at Nabble.com.
Troy A. Griffitts
2010-11-05 00:25:03 UTC
Permalink
Hey guys. We have been in talks with a number of Bible publishers (and
SBL is approaching in just over a week where we hope to again pitch our
software to even more publishers) and always point there to our wiki for
technical details of running their own module repository:

http://crosswire.org/wiki/Creating_and_Maintaining_a_Module_Repository

This page has a long history of edits by various individuals among whom
I have only participated minimally.

Is there any chance I can get some of you who have contributed to this
valuable resource to clean it up a bit.

I personally would like to see:

its size reduced as small as possible to show how easy it is to run a
remote repository. Initial, "simple repository" section greatly
appreciated, but the rest is still a little cloudy for even me to
understand.

important technical typos fixed (like mods.d.tar.g[z] fixed). Proper
English sentences (sorry non-native speakers. I do value your input).

it organized appropriately, with talks about having .zip files handy for
people in persecuted countries and for mobile users without builtin
installers (do we have any of these?) either moved to another page, as
this has nothing to do with running a remote repository, or else moved
to the end in a section explicitly delineated as not part of running a
remote repository-- again, trying to make this task look very easy. I'm
not saying this isn't valuable information, but I want these
organizations to buy in for as little work possible.

I hope you understand how this benefits the pitch.

Thanks tons for your support,

Troy
Post by David Haslam
In the KJV module, there are 17 occurences of the alternative method to mark
translation changes.
<seg subType="x-added" type="x-transChange">...</seg>This alternative method
is a hack which seems to be required when the transChange text is within a
<w> container.
I assume that OSIS does not permit the recommended
<transChange>...</transChange> container element in such circumstances.
Xiphos 3.1.3 displays these without italics. This is a bug.
I have also observed that diatheke with RTF output_format does the same, so
perhaps the bug is in the SWORD API.
See also http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach
http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach
I have also reported the bug in the Xiphos tracker. ID=3103244.
David
Troy A. Griffitts
2010-11-05 00:44:48 UTC
Permalink
I just wanted to add that...


It is a huge part of our vision to have a broad number of publishers
host their own repositories of material for use with our software. We
do not seek out license deals with publishers like other Bible software
vendors do. We offer the ability for publishers to make their resources
available with minimal work on a vast number of platforms. This is our
selling point: no fees for us, minimal effort for them to publish their
own materials.

We have spent countless hours developing our internal distributed
installation mechanism and our frontend developers have done a great job
exposing this beautifully to the end user, to work toward this becoming
a reality. It's been a long time in the works and now is finally
available in most all of our frontend applications.

Our primary goal has always been to make the Gospel of Christ as
available as possible to a lost world. Bible translations are
personally my highest priority, but publishers-- both commercial and
non-commercial-- have valuable resources for training the Body of
Christ. I would also like to see these resources become available for
our users.

Just wanted to share my heart on this matter.

Troy
Post by Troy A. Griffitts
Hey guys. We have been in talks with a number of Bible publishers (and
SBL is approaching in just over a week where we hope to again pitch our
software to even more publishers) and always point there to our wiki for
http://crosswire.org/wiki/Creating_and_Maintaining_a_Module_Repository
This page has a long history of edits by various individuals among whom
I have only participated minimally.
Is there any chance I can get some of you who have contributed to this
valuable resource to clean it up a bit.
its size reduced as small as possible to show how easy it is to run a
remote repository. Initial, "simple repository" section greatly
appreciated, but the rest is still a little cloudy for even me to
understand.
important technical typos fixed (like mods.d.tar.g[z] fixed). Proper
English sentences (sorry non-native speakers. I do value your input).
it organized appropriately, with talks about having .zip files handy for
people in persecuted countries and for mobile users without builtin
installers (do we have any of these?) either moved to another page, as
this has nothing to do with running a remote repository, or else moved
to the end in a section explicitly delineated as not part of running a
remote repository-- again, trying to make this task look very easy. I'm
not saying this isn't valuable information, but I want these
organizations to buy in for as little work possible.
I hope you understand how this benefits the pitch.
Thanks tons for your support,
Troy
Post by David Haslam
In the KJV module, there are 17 occurences of the alternative method to mark
translation changes.
<seg subType="x-added" type="x-transChange">...</seg>This alternative method
is a hack which seems to be required when the transChange text is within a
<w> container.
I assume that OSIS does not permit the recommended
<transChange>...</transChange> container element in such circumstances.
Xiphos 3.1.3 displays these without italics. This is a bug.
I have also observed that diatheke with RTF output_format does the same, so
perhaps the bug is in the SWORD API.
See also http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach
http://crosswire.org/wiki/OSIS_Bibles#Recommended_Approach
I have also reported the bug in the Xiphos tracker. ID=3103244.
David
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Matthew Talbert
2010-11-05 01:09:48 UTC
Permalink
I know this has been an area of disagreement in the past, and I'm not
intending to start any sort of argument, but just share my experience.
Among the businesses I work with (mostly small operations), there is
almost no familiarity with FTP at all, and in many cases (in the
larger ones), FTP is often blocked both incoming and outgoing. Many of
these are Windows environments with IIS for a web server. Looking at
the skills of these organizations and their IT departments, setting up
an FTP server would not be the ideal choice of deployment. Especially
when it comes to things like mods.tar.gz which require tools outside
their normal tool range. A much more ideal repository format for these
types of organizations, would be that used by jsword. It just requires
a simple HTTP-accessible directory, in which a bunch of zip files are
dropped. Ideally there would be one file with a list of what's
available.

Again, I'm not arguing what should be done, or even what is the most
common setup, but in my experience the current requirements of setting
up a repo with FTP would not be the ideal solution for people with
little IT experience. And as we have discussed before, it is difficult
to find good FTP hosting on the web, and challenging for users to set
up. It is much simpler and cheaper to find HTTP hosting.

I know we now can access repos over HTTP, but it seems to me quite
fragile. It depends on Apache's directory listing, and even running it
through Squid (or other proxy servers) completely breaks it. I haven't
tried it, but I suspect using IIS would completely break it as well,
as it has a different directory listing.

So, our repo setup guidelines are not really very simple at all, but
pre-suppose a wide variety of things like "client and server must not
be behind proxies", "client and server must have FTP ports open",
"server must be running apache for HTTP access", and so on. I'm afraid
these sort of requirements are going to cause difficulties in getting
other organizations set up, if they have IT departments and
environments anything like what I see daily.

Matthew
Troy A. Griffitts
2010-11-05 02:46:51 UTC
Permalink
I do understand the very real practical concerns about FTP you point
out, but we have practically received almost no support emails from
users not being able to install because FTP was blocked. Most hosting
services provide FTP access. The issue we ran into before was some
didn't provide anonymous ftp access, which we've rectified by adding
username / password fields in our repository registry and support for
this in our installer, and in reality to install an FTP server on a
windows box is trivial compared to the ongoing maintenance of assuring
zips are created and placed in the correct folder, an index file for the
zips is created and kept updated whenever anything changes, etc.

It might seem trivial to us, and might actually be trivial for the
publishers, in practice (and might eventually be how we end up going if
we do find real roadblocks with our current FTP mechanism), but:

It sure is easy to sell: "just get your library of books installed and
working in any of our frontends, and then point an FTP server to that
installation and you are now publishing your own remote repository; drop
that installation onto a CD or USB stick and you can use that media as
an install source with any of our frontends; drop it to a network drive
and you can use that as an install source for any of our frontends..."
(I realize the last 2 don't have anything to do with the FTP issue at
hand, but it goes along with the general idea of making it really easy
for anyone to share their library with others in most any setting)

Or, "you currently have 100+ of your Bibles formatted as SWORD modules
you are publishing from your server via SWORDWeb. All you have to do is
point an FTP server to this same data path and you are now publishing
your 100+ Bibles for use in all of our software on any of our supported
platforms: Windows, Linux, Mac, iPhone, Android, etc."


These are real world cases right now.

And the most important thing for us is to have a unified message to the
publishers, that all of our software can install from a common
repository format.

Again, I'm not amiss to requiring zips and an index file if we find it
necessary, but if it is not necessary, I would rather us do alot of
extra work to make life even the slightest bit simpler for anyone who
wishes to make their materials available.

And also, as said to the JSword team, I'm not opposed to them having
their own mechnism they feel is better, but I only ask that it is
optional-- in the sense that, failing to locate their own mechanism on a
remote repository, they fall back to the common FTP mechanism.


Hope this communicates that I understand the concerns and have
considered them and made what I hope we can at least agree is a good
solution, if not the best in your mind.

Troy





.but the fact of the matter is that FTP is the Internet protocol for
browsing a tree of files and for transferring those files
Post by Matthew Talbert
I know this has been an area of disagreement in the past, and I'm not
intending to start any sort of argument, but just share my experience.
Among the businesses I work with (mostly small operations), there is
almost no familiarity with FTP at all, and in many cases (in the
larger ones), FTP is often blocked both incoming and outgoing. Many of
these are Windows environments with IIS for a web server. Looking at
the skills of these organizations and their IT departments, setting up
an FTP server would not be the ideal choice of deployment. Especially
when it comes to things like mods.tar.gz which require tools outside
their normal tool range. A much more ideal repository format for these
types of organizations, would be that used by jsword. It just requires
a simple HTTP-accessible directory, in which a bunch of zip files are
dropped. Ideally there would be one file with a list of what's
available.
Again, I'm not arguing what should be done, or even what is the most
common setup, but in my experience the current requirements of setting
up a repo with FTP would not be the ideal solution for people with
little IT experience. And as we have discussed before, it is difficult
to find good FTP hosting on the web, and challenging for users to set
up. It is much simpler and cheaper to find HTTP hosting.
I know we now can access repos over HTTP, but it seems to me quite
fragile. It depends on Apache's directory listing, and even running it
through Squid (or other proxy servers) completely breaks it. I haven't
tried it, but I suspect using IIS would completely break it as well,
as it has a different directory listing.
So, our repo setup guidelines are not really very simple at all, but
pre-suppose a wide variety of things like "client and server must not
be behind proxies", "client and server must have FTP ports open",
"server must be running apache for HTTP access", and so on. I'm afraid
these sort of requirements are going to cause difficulties in getting
other organizations set up, if they have IT departments and
environments anything like what I see daily.
Matthew
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
admin
2010-11-05 12:40:20 UTC
Permalink
Post by Troy A. Griffitts
I do understand the very real practical concerns about FTP you point
out, but we have practically received almost no support emails from
users not being able to install because FTP was blocked. Most hosting
services provide FTP access. The issue we ran into before was some
didn't provide anonymous ftp access, which we've rectified by adding
username / password fields in our repository registry and support for
this in our installer, and in reality to install an FTP server on a
windows box is trivial compared to the ongoing maintenance of assuring
zips are created and placed in the correct folder, an index file for the
zips is created and kept updated whenever anything changes, etc.
I don't think that installing an FTP server is trivial.
I can't agree that 'most hosting services provide FTP access.'

Most cheep hosting services provide ftp access only for its maintainer,
ftp access is not permitted for public use.
HTTP services are name-based , FTP services are IP-based. So
if one want to service an FTP server, one must own one IP address
for it, contrary to the fact that many HTTP services can share one
IP address.

Furthermore FTP service needs two connections , one for control and
one for data. one is outgoing, another is incoming. this make
firewall rules very complexing.
Servicing an HTTP server is easy, and servicing an FTP server is annoying.
If both methods can serve same jobs, I would rather choose HTTP .
Post by Troy A. Griffitts
It might seem trivial to us, and might actually be trivial for the
publishers, in practice (and might eventually be how we end up going if
--
admin at bible.salterrae.net
Peter von Kaehne
2010-11-05 13:02:58 UTC
Permalink
I think Troy, the concern is correct.

For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must.

Having a multiplicity of methods of getting modules into the system would certainly be easier.

My preference:

1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp

2) add methods for local installation of zips. Look at MK Bible - pull a zip over the programme and it gets installed. This is - emphatically - not how I would want to install a large selection of modules or how I would want to publish the same, but it is probably the best usability i have seen for a frontend using only 2-3 modules (which is what MKBible is laid out for) and a small time publisher or someone who has target audience of little computer literacy

Peter
Von: admin at bible.salterrae.net
An: "SWORD Developers\' Collaboration Forum" <sword-devel at crosswire.org>
Betreff: Re: [sword-devel] Remote Module Repository Wiki
Post by Troy A. Griffitts
I do understand the very real practical concerns about FTP you point
out, but we have practically received almost no support emails from
users not being able to install because FTP was blocked. Most hosting
services provide FTP access. The issue we ran into before was some
didn't provide anonymous ftp access, which we've rectified by adding
username / password fields in our repository registry and support for
this in our installer, and in reality to install an FTP server on a
windows box is trivial compared to the ongoing maintenance of assuring
zips are created and placed in the correct folder, an index file for the
zips is created and kept updated whenever anything changes, etc.
I don't think that installing an FTP server is trivial.
I can't agree that 'most hosting services provide FTP access.'
Most cheep hosting services provide ftp access only for its maintainer,
ftp access is not permitted for public use.
HTTP services are name-based , FTP services are IP-based. So
if one want to service an FTP server, one must own one IP address
for it, contrary to the fact that many HTTP services can share one
IP address.
Furthermore FTP service needs two connections , one for control and
one for data. one is outgoing, another is incoming. this make
firewall rules very complexing.
Servicing an HTTP server is easy, and servicing an FTP server is annoying.
If both methods can serve same jobs, I would rather choose HTTP .
Post by Troy A. Griffitts
It might seem trivial to us, and might actually be trivial for the
publishers, in practice (and might eventually be how we end up going if
--
admin at bible.salterrae.net
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
GMX DSL Doppel-Flat ab 19,99 &euro;/mtl.! Jetzt auch mit
gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl
Greg Hellings
2010-11-05 13:15:24 UTC
Permalink
Post by Peter von Kaehne
I think Troy, the concern is correct.
For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must.
Having a multiplicity of methods of getting modules into the system would certainly be easier.
1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp
While I very strongly agree that FTP should not be held up as the best
solution for everyone (I have worked with and for several
organizations who flat out refused to ever allow FTP of any type to be
a part of their systems and who primarily ran IIS for their web
content), I'm not sure the argument from excessive cost is accurate.

Annual DNS leasing from my provider is less than $40/year (I want to
say it's only about $17, but I might be remembering incorrectly)
Dedicated VPS hosting with dedicated IP address, about 20 GB of drive
space and full root access to my system is $19.95/mo.

If $23 per month is excessive for anyone who really intends to
distribute modules, they can almost certainly find someone else in the
SWORD community who will be willing to host it for them at very little
to no cost. Additionally I can host any low-volume traffic I want
from home by registering with a place like dyndns.com and setting up a
server on any connection that is moderately reliable (my home). The
option is not viable for people in developing countries but is
certainly more than viable for any first world country I'm aware of.

--Greg
Peter von Kaehne
2010-11-05 13:23:09 UTC
Permalink
The option is not viable for people in developing countries but is
certainly more than viable for any first world country I'm aware of.
And that is to some extent one of our main target audiences - it is nice to have excellent English language bible study material on CW, but we are the only people who provide stuff for about 100 languages or so (and growing). To make this easier and to invite contribution is my real desire.

Peter
--
Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief!
Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail
Troy A. Griffitts
2010-11-05 14:35:53 UTC
Permalink
There are a number of finely delineated topics here we should separate.

1) Data format for our "module installation source".

i.e, is it a list of zips, is it an existing install set, etc.

Simply to be explicit, we have obvious use cases which come into play:

a) Installing our software -- all implementations will be different
because we have a variety of packages running on a variety of platforms.
b) Installing modules from a module installation source
c) Running our software

The final item (c) seems out of place but really is important. We have
successfully achieved a common data storage format for an installation
set of books (i.e. "library/mods.d/..., library/modules/...) across a
vast number of platforms. Sure this hierarchy of files moves around to
different locations on different platforms, but once the location is
pinpointed, everything under that location remains the same for any of
our platforms.

The 'holy grail' for implementation of the (1) above is for both (b) and
(c) use cases to be handled by the SAME dataset, and is what we are
working towards. In other words, if an installed module library can,
itself, also be an installation source for other instances of any SWORD
software, then distribution of Bibles becomes viral. This brings us to
topic 2.

2) The viral ability END USERS to share modules.

If I can, for example,

o use the builtin installer of my SWORD software on my Android device to
install my important modules
o then plug it into my friends computer and install modules FROM my
android device WITHOUT ANY PREPARATION
o then from his computer right-click on the top module library folder
and "Share as... 'SWORD Modules'",
o then go to any computer on the network and install modules from that
"SWORD Modules" share,
o then install FileZilla on one of those computers and let anyone over
the Internet install Bibles from that location,
o then point IIS or Apache to that location (http transport needs work
but is on the way-- more talk below)

... distribution of Bibles then becomes viral. It is spontaneous. No
preparation. I have my library of books I use regularly with me and if
someone wants one of my books, then they can install straight from my
library.


3) The TRANSPORT TECHNOLOGY used to enable this over the Internet.

We chose FTP because it was more practical. It is very difficult to
implement the programmatic traversal of a directory and file hierarchy
over HTTP. Apache and IIS all spew out different html for directory
listings.

Now having said this, Nic Carter submitted the first pass at this which
works for our apache server (and likely many apache servers which spew
out the vanilla html index page like here:
http://crosswire.org/ftpmirror/pub/

So, HTTP is currently possible in the engine and if we get push back
from publishers or find users not able to use FTP due to firewall
restrictions, we plan to fully support HTTP moving forward. We simply
need to break out our directory parsing code and make it work well with
a few of the more mainstream httpd services like Apache and IIS.


4) A single zip file for installation of a module (or modules) is simply
a zip of (1c) above.

Of the 3 'zip' formats we support for download from our server, only the
'rawzip' format should survive. I believe I can safely say we all agree
on that. This is in essence still the same thing. It is a module
repository of exactly 1 module in a difference storage medium. Heck it
could be 5 installed modules all zipped up and it would still be the
same thing. As I see it, it is an end user preference left to the
exercise of frontend developers. "Can my users operate their file
browser and drag and drop to my app easier than they can navigate their
"Browse to folder" dialog box in "Install SWORD Modules from path..."
option in my installer? If so, then I should support drag and drop on
my frontend-- after all it is simply an unzip. But the user gets no
benefit of our full installer: not description, to versioning upgrade
information if they already have an instance of that module installed,
to category, no language, etc.

5) If we find we cannot achieve this 'no prep' installation utopia in
the real world and it is necessary to have an intermediate format.
Matthew Talbert had a great suggestion on IRC last night to simply add a
'prep for publishing' (or in his words simply "publish") option to the
InstallMgr interface which would create (1) "module installation source"
data format if indeed it technically must be different. Great idea if
we need to concede to 2 disparate data formats, but I am still hopeful
we can avoid this extra step for would-be module distributors-- end
users and publishers alike.


Hope this separates out thoughts a little better.


Troy
Post by Greg Hellings
Post by Peter von Kaehne
I think Troy, the concern is correct.
For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must.
Having a multiplicity of methods of getting modules into the system would certainly be easier.
1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp
While I very strongly agree that FTP should not be held up as the best
solution for everyone (I have worked with and for several
organizations who flat out refused to ever allow FTP of any type to be
a part of their systems and who primarily ran IIS for their web
content), I'm not sure the argument from excessive cost is accurate.
Annual DNS leasing from my provider is less than $40/year (I want to
say it's only about $17, but I might be remembering incorrectly)
Dedicated VPS hosting with dedicated IP address, about 20 GB of drive
space and full root access to my system is $19.95/mo.
If $23 per month is excessive for anyone who really intends to
distribute modules, they can almost certainly find someone else in the
SWORD community who will be willing to host it for them at very little
to no cost. Additionally I can host any low-volume traffic I want
from home by registering with a place like dyndns.com and setting up a
server on any connection that is moderately reliable (my home). The
option is not viable for people in developing countries but is
certainly more than viable for any first world country I'm aware of.
--Greg
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Weston Ruter
2010-11-05 14:45:08 UTC
Permalink
What about using WebDAV for HTTP? It supports directory listings (i.e.
collections).
Post by Troy A. Griffitts
3) The TRANSPORT TECHNOLOGY used to enable this over the Internet.
We chose FTP because it was more practical. It is very difficult to
implement the programmatic traversal of a directory and file hierarchy
over HTTP. Apache and IIS all spew out different html for directory
listings.
Now having said this, Nic Carter submitted the first pass at this which
works for our apache server (and likely many apache servers which spew
http://crosswire.org/ftpmirror/pub/
So, HTTP is currently possible in the engine and if we get push back
from publishers or find users not able to use FTP due to firewall
restrictions, we plan to fully support HTTP moving forward. We simply
need to break out our directory parsing code and make it work well with
a few of the more mainstream httpd services like Apache and IIS.
--
Weston Ruter
http://weston.ruter.net/
@westonruter <http://twitter.com/westonruter> - Google
Profile<http://www.google.com/profiles/WestonRuter#about>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101105/17870c8d/attachment-0001.html>
Troy A. Griffitts
2010-11-05 15:25:07 UTC
Permalink
Thanks for the suggestion Weston,

I had no idea WebDAV allowed programmatic traversal of remote content.
That's great to know!

I wonder at the value of depending on WebDAV to solve our problem though.

Our current reasons to switch to HTTP is that HTTP is ubiquitous-- often
already available at institutions, usually requires no additional
training to make something available, and has no firewall/security
concerns. I wonder if WebDAV still gives us those benefits.

Troy
Post by Weston Ruter
What about using WebDAV for HTTP? It supports directory listings (i.e.
collections).
On Fri, Nov 5, 2010 at 3:35 PM, Troy A. Griffitts <scribe at crosswire.org
3) The TRANSPORT TECHNOLOGY used to enable this over the Internet.
We chose FTP because it was more practical. It is very difficult to
implement the programmatic traversal of a directory and file hierarchy
over HTTP. Apache and IIS all spew out different html for directory
listings.
Now having said this, Nic Carter submitted the first pass at this which
works for our apache server (and likely many apache servers which spew
http://crosswire.org/ftpmirror/pub/
So, HTTP is currently possible in the engine and if we get push back
from publishers or find users not able to use FTP due to firewall
restrictions, we plan to fully support HTTP moving forward. We simply
need to break out our directory parsing code and make it work well with
a few of the more mainstream httpd services like Apache and IIS.
--
Weston Ruter
http://weston.ruter.net/
@westonruter <http://twitter.com/westonruter> - Google Profile
<http://www.google.com/profiles/WestonRuter#about>
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Greg Hellings
2010-11-05 15:38:30 UTC
Permalink
Post by Troy A. Griffitts
Thanks for the suggestion Weston,
I had no idea WebDAV allowed programmatic traversal of remote content.
That's great to know!
I wonder at the value of depending on WebDAV to solve our problem though.
Every hosting provider I know of supports it - though I'm sure some
charge it for higher prices than their base entry fee.

Linux natively supports it all over the place for client access.
Windows has been able to mount WebDAV natively as a remote location
since at least Windows 98.
Mac OS X supports WebDAV since 10.0

Apache has pretty straightforward support for it, and IIRC, IIS has
support that's about as easy to enable as flipping a switch. If
someone can setup a webserver, they can setup WebDAV.

--Greg
David Overcash
2010-11-05 15:51:05 UTC
Permalink
Post by Troy A. Griffitts
Our current reasons to switch to HTTP is that HTTP is ubiquitous-- often
already available at institutions, usually requires no additional
training to make something available, and has no firewall/security
concerns. I wonder if WebDAV still gives us those benefits.
Sorry - coming into this a bit late... but it seems to me that HTTP would
make the most sense when bundled with some standard we created for directory
listings, modules, etc. In addition to being widely supported, it is also
easy to use HTTPS to create simple-encryption in addition to it being easily
accessed through just about any proxy service (if necessary).

I know that it would make much more sense to use HTTP when it comes to
mobile platforms (speaking from experience).

-David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101105/85d9c404/attachment-0001.html>
Butrus Damaskus
2010-11-08 21:14:46 UTC
Permalink
Post by David Overcash
Post by Troy A. Griffitts
Our current reasons to switch to HTTP is that HTTP is ubiquitous-- often
already available at institutions, usually requires no additional
training to make something available, and has no firewall/security
concerns. ?I wonder if WebDAV still gives us those benefits.
Sorry - coming into this a bit late... but it seems to me that HTTP would
make the most sense when bundled with some standard we created for directory
listings, modules, etc. ?In addition to being widely supported, it is also
easy to use HTTPS to create simple-encryption in addition to it being easily
accessed through just about any proxy service (if necessary).
I know that it would make much more sense to use HTTP when it comes to
mobile platforms (speaking from experience).
-David
a) WebDAV is a _SUPERSET_ of HTTP. So HTTP GET, HEAD, PUT, MOVE, COPY
methods are used + they are extenden with other methods (like PROPFIND
for listing directory / metadata fetching)

b) WebDAV of course supports TLS/SSL
DM Smith
2010-11-05 14:55:00 UTC
Permalink
Post by Troy A. Griffitts
2) The viral ability END USERS to share modules.
Don't we have to be concerned with locked modules? With the unlock
key/cipher being held in the conf, isn't that a bad thing?

Should we do as BibleTime and have a separate store of keys? If so, where?

In Him,
DM
Troy A. Griffitts
2010-11-05 14:58:26 UTC
Permalink
I thought of exactly this same thing when writing that! :)

Yes.
Post by DM Smith
Post by Troy A. Griffitts
2) The viral ability END USERS to share modules.
Don't we have to be concerned with locked modules? With the unlock
key/cipher being held in the conf, isn't that a bad thing?
Should we do as BibleTime and have a separate store of keys? If so, where?
In Him,
DM
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Karl Kleinpaste
2010-11-05 15:09:52 UTC
Permalink
Post by Troy A. Griffitts
Matthew Talbert had a great suggestion on IRC last night to simply add
a 'prep for publishing' (or in his words simply "publish") option to
the InstallMgr interface...
It's great, but funny, to see all the sudden interest in publishing
users' content. I've talked about it a couple times here in
sword-devel, but the idea never got any traction before. Why now?

Nearly 3 years ago:
http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html
Troy A. Griffitts
2010-11-05 15:19:24 UTC
Permalink
I don't consider End Users Publishing and Publishing End Users the same
thing :)

The "Publish" option would simply enable the installed set of modules to
be 'transformed' into the hypothetical install source format, if we ever
decide we need a different install source format.

For example, if I install KJV, AraSVD, MHC, StrongsRealGreek,
StrongsRealHebrew, Robinsons, and TR on my device, and decide to share
my books with someone else, I would have to first press a "Publish"
button or some such to generate the format necessary (if a different
format ever becomes necessary) for another instance of a SWORD
application to install from my books.

I believe your idea was for allowing users to publish their authored
content to a publicly recognized location for discovery by other users.

I don't think I ever voiced disapproval to that idea. I just didn't
want to host (police) their content :)

Troy
Post by Karl Kleinpaste
Post by Troy A. Griffitts
Matthew Talbert had a great suggestion on IRC last night to simply add
a 'prep for publishing' (or in his words simply "publish") option to
the InstallMgr interface...
It's great, but funny, to see all the sudden interest in publishing
users' content. I've talked about it a couple times here in
sword-devel, but the idea never got any traction before. Why now?
http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Karl Kleinpaste
2010-11-05 16:15:39 UTC
Permalink
Post by Troy A. Griffitts
I don't consider End Users Publishing and Publishing End Users the
same thing :)
Well... I suppose I consider them alternate cases of the same
underlying technical capability: "I've got content that you can get."
Post by Troy A. Griffitts
I believe your idea was for allowing users to publish their authored
content to a publicly recognized location for discovery by other
users. I don't think I ever voiced disapproval to that idea. I just
didn't want to host (police) their content :)
Users' self-authored content was a specific application, because we've
had users asking about that since before I became involved with then-
GnomeSword. People want to write commentaries (of a sort), Bible study
outlines, prayer journals, and other such stuff, and want to share it.
Analogize to bible.org's efforts at distributing lots of theologically-
useful content. The general problem is, of course, getting _any_
user-held content, however obtained or created, into an accessible place
and format for others to retrieve.

As DM has just mentioned, and as I was wondering obliquely in that note
from 3 yrs ago, the biggest hassle is how to prevent restricted content
from going viral. But in general, there is the more precise question,
"How do I make *only these N modules* (out of my entire library)
available?", because the user may not want (e.g.) his personal journals
to become part of the set made available. "Restricted content" can take
many forms: If someone has been keeping a prayer log about family
problems, he needs assurance that his _/entire/_ module set is not exposed.

The particular problem for how to store and use crypt keys outside of
their ("natural"?) xyz.conf will need to be addressed. Xiphos stores
them that way, and I thought most apps did so, but I suppose we should
see what code from BibleTime we could borrow so that keys don't need to
remain stored that way.
DM Smith
2010-11-05 16:35:08 UTC
Permalink
Post by Karl Kleinpaste
The particular problem for how to store and use crypt keys outside of
their ("natural"?) xyz.conf will need to be addressed. Xiphos stores
them that way, and I thought most apps did so, but I suppose we should
see what code from BibleTime we could borrow so that keys don't need to
remain stored that way.
Just a couple of thoughts:
IIRC, multiple confs can be slurped into a multi-map. If so...
Have a separate *.conf per module of the form:
[NAME]
cipher=xxxx
Where *.conf is the same name as what is in mods.d and [NAME] matches
that in that conf.

Don't store it next to mods.d and modules. As grabbing ~/.sword (or
whatever it's called and where-ever it is located) shouldn't also grab
the keys.

For lack of a better name have a sibling dir to ~/.sword, say ~/.cwkeys
that holds them.

But if not separate from ~/.sword, then in ~/.sword/keys.d?

In Him,
DM
Troy A. Griffitts
2010-11-05 18:06:03 UTC
Permalink
Yeah,

SWConfig::operator +=

In BibleCS we store our font preferences in something like:

./userprefs.conf

The file looks something like:

[KJV]
Font=Time New Roman

[TR]
Font=SILGentium


Then we basically do

SWConfig userPrefs("./userprefs.conf");
(*swMgr->config) += userPrefs;


So, we could just as easily add CipherKey= lines in ./userprefs.conf


The trick is we need to augment the SWMgr config AFTER the config is
loaded and BEFORE SWMgr creates the SWModule object, so the cipherkey is
used during module construction.

Not sure exactly where to place it. I'll try to remember to look for a
good place and repost.
Post by DM Smith
Post by Karl Kleinpaste
The particular problem for how to store and use crypt keys outside of
their ("natural"?) xyz.conf will need to be addressed. Xiphos stores
them that way, and I thought most apps did so, but I suppose we should
see what code from BibleTime we could borrow so that keys don't need to
remain stored that way.
IIRC, multiple confs can be slurped into a multi-map. If so...
[NAME]
cipher=xxxx
Where *.conf is the same name as what is in mods.d and [NAME] matches
that in that conf.
Don't store it next to mods.d and modules. As grabbing ~/.sword (or
whatever it's called and where-ever it is located) shouldn't also grab
the keys.
For lack of a better name have a sibling dir to ~/.sword, say ~/.cwkeys
that holds them.
But if not separate from ~/.sword, then in ~/.sword/keys.d?
In Him,
DM
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Matthew Talbert
2010-11-05 21:44:57 UTC
Permalink
Post by Karl Kleinpaste
Post by Troy A. Griffitts
Matthew Talbert had a great suggestion on IRC last night to simply add
a 'prep for publishing' (or in his words simply "publish") option to
the InstallMgr interface...
It's great, but funny, to see all the sudden interest in publishing
users' content. ?I've talked about it a couple times here in
sword-devel, but the idea never got any traction before. ?Why now?
http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html
Part of this concept of "publish" was not really pushing to a remote
repository, but the idea that we could (say using the Xiphos module
manager) pick a subset of installed modules, and prepare them for
remote access. If, for example, we were to agree that an HTTP
repository could consist of a single file listing the contents
(really, no different than mods.tar.gz) + a set of zipped files, then
this "publish" would create this whole directory, ready for simply
copying to any http-accessible location.

The other part of the publish would be publishing a single module to a
remote location where a user has an account. This would be very
similar, but it would just be pushing a single zip file, and the
remote server would be responsible for putting it in the correct
location, and updating the repository listing.

Part of why I'd like very much to see a repository able to consist of
this directory listing + zipped modules is that it makes the whole
idea of remote publishing much easier. With FTP, it's a pain to set up
various directories with different permission schemes on the fly,
while using HTTP would allow typical server-side software (eg, PHP,
django, etc) to handle rather complex schemes easily. It would be
relatively simple on the same server to have a system that allowed a
user to come sign up for an account, and suddenly that user has
publish capability unique to him. Then, if he wanted to publish to a
more shared and publicly visible location, a simple flip of the switch
would allow him to publish there as well. This would allow everyone to
quickly share with their friends, but require interaction with a
repository owner before being allowed to publish there.

Matthew
Greg Hellings
2010-11-05 22:05:57 UTC
Permalink
A year or so ago I wrote the shell of a (PHP) website which would allow a
user to browse various selected modules, "purchase" one, create a unique
encryption key for that user and encrypt the module for them.

It would then give the user a single download which could be installed with
a Python/Qt app I wrote, which would also accept the user's encryption key
and enter it into the module conf file.

It never went anywhere because of various non-technical reasons, but adding
user upload and giving a user the chance to release their own modules, etc,
would be easy extra functionality.

Note: everything about the site was a temporary hack designed to prove the
technical aspects could be done and not to actually be used. E.G. It did not
actually require money and there was no user sign up and the like. But those
are trivial and off topic from point of view of the SWORD functionality.

--Greg
Post by Matthew Talbert
Post by Karl Kleinpaste
Post by Troy A. Griffitts
Matthew Talbert had a great suggestion on IRC last night to simply add
a 'prep for publishing' (or in his words simply "publish") option to
the InstallMgr interface...
It's great, but funny, to see all the sudden interest in publishing
users' content. I've talked about it a couple times here in
sword-devel, but the idea never got any traction before. Why now?
http://www.crosswire.org/pipermail/sword-devel/2008-January/026736.html
Part of this concept of "publish" was not really pushing to a remote
repository, but the idea that we could (say using the Xiphos module
manager) pick a subset of installed modules, and prepare them for
remote access. If, for example, we were to agree that an HTTP
repository could consist of a single file listing the contents
(really, no different than mods.tar.gz) + a set of zipped files, then
this "publish" would create this whole directory, ready for simply
copying to any http-accessible location.
The other part of the publish would be publishing a single module to a
remote location where a user has an account. This would be very
similar, but it would just be pushing a single zip file, and the
remote server would be responsible for putting it in the correct
location, and updating the repository listing.
Part of why I'd like very much to see a repository able to consist of
this directory listing + zipped modules is that it makes the whole
idea of remote publishing much easier. With FTP, it's a pain to set up
various directories with different permission schemes on the fly,
while using HTTP would allow typical server-side software (eg, PHP,
django, etc) to handle rather complex schemes easily. It would be
relatively simple on the same server to have a system that allowed a
user to come sign up for an account, and suddenly that user has
publish capability unique to him. Then, if he wanted to publish to a
more shared and publicly visible location, a simple flip of the switch
would allow him to publish there as well. This would allow everyone to
quickly share with their friends, but require interaction with a
repository owner before being allowed to publish there.
Matthew
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101105/e43f517a/attachment.html>
Nic Carter
2010-11-06 04:48:20 UTC
Permalink
Post by Troy A. Griffitts
2) The viral ability END USERS to share modules.
If I can, for example,
o use the builtin installer of my SWORD software on my Android device to
install my important modules
o then plug it into my friends computer and install modules FROM my
android device WITHOUT ANY PREPARATION
o then from his computer right-click on the top module library folder
and "Share as... 'SWORD Modules'",
o then go to any computer on the network and install modules from that
"SWORD Modules" share,
o then install FileZilla on one of those computers and let anyone over
the Internet install Bibles from that location,
o then point IIS or Apache to that location (http transport needs work
but is on the way-- more talk below)
... distribution of Bibles then becomes viral. It is spontaneous. No
preparation. I have my library of books I use regularly with me and if
someone wants one of my books, then they can install straight from my
library.
I don't think this should hold back usability in other ways. I would rather that we provided some other functionality (or tool) to make this easy for end users to do, rather than make this mean that the final install of a module is a complete install source. I love the concept, but I think that your example is something that 0.0001% of users would be interested in, given the nerdery required to know how to do that! ;)
BUT
I liked the suggestion made in another post of having a "prepare for distribution" or "publish" menu item that would then automatically prepare a module (or selection of modules) for distribution. :) Use that menu item as step one, and then you can proceed through that list of examples that Troy has. As an added extra bonus, you get to specify the location to which to "Save" the module/s ready for distribution. That way a user doesn't even need to know where her/his modules are installed to, but instead can say "desktop" (yes, I'm using the LCD of a user who does everything on their desktop!) and then easily find that location & "share" that folder. :) (or it could be specified as a network drive, for easy access to everyone on the network, etc.)

Anyway, just some more thoughts. Sorry they're coming so much later than when these were posted -- a fun consequence of living in Oz. :)

ps: I could easily be persuaded to add an "export" feature to PocketSword, which would work over WiFi or 3G (similar to the current way of manually installing a module from a ZIP file), so you could share modules on your device with friends. Of course, this would need to be switched on manually, and wouldn't always be running in the background, so as to conserve battery! But would be quite funky. :) But given you need to press a button to switch it on, at that point I could get PocketSword to "prep" the selected modules for distribution. :)
Troy A. Griffitts
2010-11-06 11:42:08 UTC
Permalink
Hi Nic,

Thanks for the feedback. I'd like some clarification though. I really
am trying to make an honest assessment of where we are and need to go in
the future and note the negative feedback from frontend developers, but
haven't been able to quantify what exactly that negative feedback is.
Help me understand your exact objections and how an alternative scheme
Post by Nic Carter
I was going to download the mods.d.tar.gz file (over HTTP) and use
that for moduleinformation. Then download the ZIP of the module
over HTTP. If a repo doesn't have the zip folder (or the
mods.d.tar.gz file), then I would either have a fall-back to use the
normal InstallManager, or perhaps I'd just say that PocketSword
wouldn't support that repo. But after a year of PocketSword doing
things the "right" way, I've decided to take the JSword approach and
hope that doesn't cause too many issues... :(
:)

OK, This is vital for me to understand your experience. I presume the
"right" way you speak of above is simply using the InstallMgr API.

Why are you unhappy with the InstallMgr API? Does it not work? Is it
too hard to use? You mention speed, below...
Post by Nic Carter
my personal tests on an iOS device show that accessing the CrossWire
repo via FTP is reasonably slower than accessing it via HTTP.
Really? I wouldn't have expected this at all.
Post by Nic Carter
I want the reasonably large performance gains that I can gain from
simply switching protocols! :)
Yes, indeed. This is a very valid concern. Let me ask, what FTP plugin
were you using, ftplib or curl? I am wondering if it might be the
difference in plugin. I recently switched BibleCS's InstallMgr GUI to
use the ftplib system instead of CURL and surprisingly seem to have less
troubles and more responsiveness. There is still one major shortcoming
of ftplib-- it cannot download directly to a buffer for a directory
listing, we use a temporary file-- but I hope to resolve that in the
next day or two.

Speed is a very real and practical concern and if we have a problem, we
need to resolve it.
Post by Nic Carter
Post by Troy A. Griffitts
2) The viral ability END USERS to share modules.
If I can, for example,
o use the builtin installer of my SWORD software on my Android
device to install my important modules o then plug it into my
friends computer and install modules FROM my android device WITHOUT
ANY PREPARATION o then from his computer right-click on the top
module library folder and "Share as... 'SWORD Modules'", o then go
to any computer on the network and install modules from that "SWORD
Modules" share, o then install FileZilla on one of those computers
and let anyone over the Internet install Bibles from that location,
o then point IIS or Apache to that location (http transport needs
work but is on the way-- more talk below)
... distribution of Bibles then becomes viral. It is spontaneous.
No preparation. I have my library of books I use regularly with
me and if someone wants one of my books, then they can install
straight from my library.
I don't think this should hold back usability in other ways.
In what way does usability relate to storage format or network transport?
Post by Nic Carter
I would rather that we provided some other functionality (or tool) to
make this easy for end users to do, rather than make this mean that
the final install of a module is a complete install source. I love
the concept, but I think that your example is something that 0.0001%
of users would be interested in, given the nerdery required to know
how to do that! ;)
I don't understand the suggestion or the "nerdiness" of the current
method. We (the engine) does provide install functionality and we
(frontend developers) do provide a tool that makes this easy for end
users. Users already know how to install modules in their favorite
software. The user wishing to install follows their familiar procedure
to install. When prompted from which Remote Repository they would like
to install, the user simply clicks a 'browse to local folder' button. I
believe most frontends already support this for installing from a CD.
Post by Nic Carter
BUT I liked the suggestion made in another post of having a "prepare
for distribution" or "publish" menu item that would then
automatically prepare a module (or selection of modules) for
distribution. :) Use that menu item as step one, and then you can
proceed through that list of examples that Troy has. As an added
extra bonus, you get to specify the location to which to "Save" the
module/s ready for distribution. That way a user doesn't even need
to know where her/his modules are installed to, but instead can say
"desktop" (yes, I'm using the LCD of a user who does everything on
their desktop!) and then easily find that location & "share" that
folder. :) (or it could be specified as a network drive, for easy
access to everyone on the network, etc.)
The "prepare for distribution" is currently unnecessary. I hope I'm not
being daft and missing something, but how can adding a step improve user
experience? unless...

Others have voiced that users might not want to expose their entire
module set, so I like your idea slightly modified to "Install Subset of
Books To ..."feature for these users, but for your objection, a simple
"Show library location" or even "Open library location" which would
launch their file explorer to the library path, would be cool.
Post by Nic Carter
Anyway, just some more thoughts. Sorry they're coming so much later
than when these were posted -- a fun consequence of living in Oz.
:)
No, no! They are extremely valuable. I need to hear real world
feedback on what works and what doesn't. What is hard and what makes
your life easy. Please keep the feedback coming.
Post by Nic Carter
I could easily be persuaded to add an "export" feature to
PocketSword, which would work over WiFi or 3G (similar to the current
way of manually installing a module from a ZIP file), so you could
share modules on your device with friends. Of course, this would
need to be switched on manually, and wouldn't always be running in
the background, so as to conserve battery! But would be quite funky.
:) But given you need to press a button to switch it on, at that
point I could get PocketSword to "prep" the selected modules for
distribution. :)
When you plug in an iphone, is the location where pocketsword installs
modules browseable from the hosting computer as USB Storage? If so, you
already have the feature.

Are you suggesting that it takes a nerd to know where to browse to find
the books? I can understand that. And to solve that problem, I could
see an 'export' function be useful. But wouldn't it be more useful to
have something like "Show PocketSword Library Location". This solution
would prevent the unnecessary duplication of the entire library of books
simply because the installing user doesn't know where to browse.

Thanks again for the conversation. Anyone who's made it this far in
this email, you must really care about the topic! :)

Troy
Nic Carter
2010-11-07 04:15:02 UTC
Permalink
Post by Troy A. Griffitts
Post by Nic Carter
I was going to download the mods.d.tar.gz file (over HTTP) and use
that for moduleinformation. Then download the ZIP of the module
over HTTP. If a repo doesn't have the zip folder (or the
mods.d.tar.gz file), then I would either have a fall-back to use the
normal InstallManager, or perhaps I'd just say that PocketSword
wouldn't support that repo. But after a year of PocketSword doing
things the "right" way, I've decided to take the JSword approach and
hope that doesn't cause too many issues... :(
:)
OK, This is vital for me to understand your experience. I presume the
"right" way you speak of above is simply using the InstallMgr API.
Why are you unhappy with the InstallMgr API? Does it not work? Is it
too hard to use? You mention speed, below...
The "right" way is by directly plugging into the InstallMgr API and getting it to do all the work. :) In order to get it to work, I've made a couple of rather minor changes, but that's not an issue at all, and I'm very happy to maintain my own set of patches required to get the SWORD lib working properly on an iOS device. :) (or, as an aside, I could submit some patches that have #defines for iOS, but I'm not too fussed. Perhaps they would be of interest if someone wanted to use the C++ lib for Android, but I'm not sure.)

I want to switch to using HTTP only in PocketSword. Mobile phone carriers are rather horrid to deal with. My old carrier here in Australia had transparent proxies and all sorts of horrid things! Other carriers do other evil things, as well. And I am fairly certain some of them don't have FTP in the common use-set scenario for their users, even in this day and age of wireless internet, and so don't care if it works very well or not... :( I have observed first hand some annoying things with different carriers here -- so perhaps the US is ok where you only have one carrier for the iPhone & hence only one set of horridness to deal with? ;) But, HTTP is so commonplace that it's harder to break it. I have yet to add proper proxy support into PocketSword, but that is high up on the list. But given that I'm thinking about rewriting the download code for PocketSword, I've put it off and I'll do both at once?
BTW, I think the InstallMgr API works fairly well and is extremely easy to use. :) When I started work on PocketSword (I was helping Ian out at first), I tackled the downloading of modules as my first task and I found it fairly easy to figure out how to get it to work. :)
Post by Troy A. Griffitts
Post by Nic Carter
my personal tests on an iOS device show that accessing the CrossWire
repo via FTP is reasonably slower than accessing it via HTTP.
Really? I wouldn't have expected this at all.
This may have to do with phone carriers? altho, I'm fairly certain I tested this over WiFi as well... In my experience, it feels like the "internet" is moving away from FTP and using HTTP for everything? Just like HTML is now being used for many different things that it was never intended to be used for... ;) But, firewalls and proxies often screw with FTP much more than HTTP and many orgs/companies simply block FTP, hence people moving to HTTP to get around those issues. :(
Post by Troy A. Griffitts
Post by Nic Carter
I want the reasonably large performance gains that I can gain from
simply switching protocols! :)
Yes, indeed. This is a very valid concern. Let me ask, what FTP plugin
were you using, ftplib or curl? I am wondering if it might be the
difference in plugin. I recently switched BibleCS's InstallMgr GUI to
use the ftplib system instead of CURL and surprisingly seem to have less
troubles and more responsiveness. There is still one major shortcoming
of ftplib-- it cannot download directly to a buffer for a directory
listing, we use a temporary file-- but I hope to resolve that in the
next day or two.
Speed is a very real and practical concern and if we have a problem, we
need to resolve it.
I see you just committed that change. :) I have only tested with curl -- as that is the only way to get HTTP downloading working. :) I never had the option to use ftplib, as downloading to a temporary file under iOS is NOT a good idea... :(
I could test with ftplib, but given the other objections I have to FTP (*sigh* phone carriers *sigh*), I'm hoping to avoid FTP. :(
Post by Troy A. Griffitts
Post by Nic Carter
Post by Troy A. Griffitts
2) The viral ability END USERS to share modules.
If I can, for example,
o use the builtin installer of my SWORD software on my Android
device to install my important modules o then plug it into my
friends computer and install modules FROM my android device WITHOUT
ANY PREPARATION o then from his computer right-click on the top
module library folder and "Share as... 'SWORD Modules'", o then go
to any computer on the network and install modules from that "SWORD
Modules" share, o then install FileZilla on one of those computers
and let anyone over the Internet install Bibles from that location,
o then point IIS or Apache to that location (http transport needs
work but is on the way-- more talk below)
... distribution of Bibles then becomes viral. It is spontaneous.
No preparation. I have my library of books I use regularly with
me and if someone wants one of my books, then they can install
straight from my library.
I don't think this should hold back usability in other ways.
In what way does usability relate to storage format or network transport?
Ok, usability may be the wrong word? But, making things less complicated in order to avoid download issues is probably a very good thing. And I consider parsing a file listing via HTTP a "more complicated" thing. I'm happy for us to do more work (like creating a "prepare for publishing" method) in order to try to make things work in more cases (ie: other webservers dishing out their own version of the file listing). If all we need to do is download 2 files, mods.d.tar.gz & then ESV.zip (as an example), that reduces places where things can go wrong... :)
Does that make sense? So, not "usability" but instead reliability? Perhaps it has something to do with my lack of trust of webservers dishing things up properly all the time & lack of speed of download on some phone carriers, causing timeouts and stuff like that? :(
Of course, perhaps many of my objections are relevant for mobile devices only and not really for other computing devices? ;)
Post by Troy A. Griffitts
Post by Nic Carter
I would rather that we provided some other functionality (or tool) to
make this easy for end users to do, rather than make this mean that
the final install of a module is a complete install source. I love
the concept, but I think that your example is something that 0.0001%
of users would be interested in, given the nerdery required to know
how to do that! ;)
I don't understand the suggestion or the "nerdiness" of the current
method. We (the engine) does provide install functionality and we
(frontend developers) do provide a tool that makes this easy for end
users. Users already know how to install modules in their favorite
software. The user wishing to install follows their familiar procedure
to install. When prompted from which Remote Repository they would like
to install, the user simply clicks a 'browse to local folder' button. I
believe most frontends already support this for installing from a CD.
Hmmm, ok, point taken. I just wonder how many users know they can do this? Surely it's easier to simply re-download a module from the repo it comes from? Or in countries where they don't want to be connecting to a "christian" server, they could use the "publish..." command? Perhaps by implementing a "publish..." command, more users would know that they are able to quickly and easily share their modules? :)
Post by Troy A. Griffitts
Post by Nic Carter
BUT I liked the suggestion made in another post of having a "prepare
for distribution" or "publish" menu item that would then
automatically prepare a module (or selection of modules) for
distribution. :) Use that menu item as step one, and then you can
proceed through that list of examples that Troy has. As an added
extra bonus, you get to specify the location to which to "Save" the
module/s ready for distribution. That way a user doesn't even need
to know where her/his modules are installed to, but instead can say
"desktop" (yes, I'm using the LCD of a user who does everything on
their desktop!) and then easily find that location & "share" that
folder. :) (or it could be specified as a network drive, for easy
access to everyone on the network, etc.)
The "prepare for distribution" is currently unnecessary. I hope I'm not
being daft and missing something, but how can adding a step improve user
experience? unless...
hehehe -- ok, good point. :) But not knowing that you can share modules would eliminate user experience of that altogether! ;)
Post by Troy A. Griffitts
Others have voiced that users might not want to expose their entire
module set, so I like your idea slightly modified to "Install Subset of
Books To ..."feature for these users, but for your objection, a simple
"Show library location" or even "Open library location" which would
launch their file explorer to the library path, would be cool.
Yes, that would be cool, and I think that would improve the user experience. But, perhaps it's simply educating users they can actually do this? But, again, I'd think that it's often quicker and easier to simply get the other user to download the module themselves from the original repo? But, yes, the persecuted church argument means we really want sharing to be possible. :)
Post by Troy A. Griffitts
Post by Nic Carter
I could easily be persuaded to add an "export" feature to
PocketSword, which would work over WiFi or 3G (similar to the current
way of manually installing a module from a ZIP file), so you could
share modules on your device with friends. Of course, this would
need to be switched on manually, and wouldn't always be running in
the background, so as to conserve battery! But would be quite funky.
:) But given you need to press a button to switch it on, at that
point I could get PocketSword to "prep" the selected modules for
distribution. :)
When you plug in an iphone, is the location where pocketsword installs
modules browseable from the hosting computer as USB Storage? If so, you
already have the feature.
Are you suggesting that it takes a nerd to know where to browse to find
the books? I can understand that. And to solve that problem, I could
see an 'export' function be useful. But wouldn't it be more useful to
have something like "Show PocketSword Library Location". This solution
would prevent the unnecessary duplication of the entire library of books
simply because the installing user doesn't know where to browse.
Ahh, the iPhone doesn't work like that. :) Apple sandbox each and every app you install, so for eg: I have PocketSword installed on my laptop in the iOS simulator to:
114:758A718D-5535-4DB4-B8E5-AD27403F1F06 nicc$ pwd
/Users/nicc/Library/Application Support/iPhone Simulator/4.1/Applications/758A718D-5535-4DB4-B8E5-AD27403F1F06

On the actual device, the equivalent would be something similarly a mess... :D
And that random string is random and you can't read or write to that folder... So, you either jailbreak and get access, or I have to write code in PocketSword to expose the modules. :)
Post by Troy A. Griffitts
Thanks again for the conversation. Anyone who's made it this far in
this email, you must really care about the topic! :)
I hope I make sense? I am a native English speaker (as much as you consider an Australian to be a native English speaker!), but sometimes don't communicate things as clearly as I should, especially as I often only get time to write nice long emails at various interesting times of the day... ;)

To summarise, I think HTTP is the direction we should go, as firewalls and (transparent?) proxies can make life hard for FTP. People work hard to make HTTP work anywhere and everywhere, as everyone wants to jump on to Google and complain if they can't, whereas there is much less demand for FTP now-a-days. :( This may be made worse by the fact that I am spending a lot of my time dealing with (silly) mobile phone carriers! And given the issues we have with HTTP file parsing, I think this provides a good time to look into the possibility of rethinking how we have repos set up. If we provided tools in order to initialise a repo, that would hopefully make things easier for Publishers to have their own repos. :D We want things to be easy for both end users and for repo maintainers, so it's important that whatever we do, life is easy for both groups. :)

Let me know if there's anything else you think needs elaboration on? Unfortunately I am mostly thinking of myself and PocketSword, so I understand if people disagree with what I've said. :)


Thanks, ybic
nic... :)
Jonathan Morgan
2010-11-07 14:28:31 UTC
Permalink
A lot here depends on evaluation of pros and cons. I personally support
HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
list of all the books and where to find them for at least some of the
following reasons. I will try and capture why succinctly:
For ZIPs:
1. Minimising the number of GETs issued (and potentially file size and
download speed): I think this is particularly important for loading the
initial list of modules, which should preferably be as fast as it possibly
can be. It also makes a difference for image modules like Dore's Woodcuts
with lots of loose files (which I remember taking a very long time to
install with Install Manager).
2. Provides one unambiguous URL for a module.
3. Uses the same format for distribution stand-alone and in InstallManager.
4. Removes most/all of the directory traversal problems involved with the
installed book structure: This has potential advantages of speed and working
with a greater number of servers as well as programmability.

For HTTP:
1. HTTP is a more common setup than FTP. Thus it is probably more
accessible to a wider group of publishers (and quite possibly users as
well). For example, it would certainly be possible to host ZIP files on
website / file sharing services such as Google Sites or DropBox, and even a
mods.d.tar.gz (though keeping it up to date could be a bit troublesome).
FTP support is a whole different kettle of fish.

I understand you want to make things easier for publishers (and that is not
a bad goal), but for me the "holy grail" is not using the same format, but
providing the best user experience. Speed is an important part of that, and
to me that almost guarantees two separate format, since the best format for
transferring over a network is almost guaranteed to be different from the
best one for reading on disk. Philosophically, I think it is usually better
to put more work on publishers rather than on consumers, because there will
usually be a lot more consumers than publishers. [I do think supporting
filesystem installs from the directory structure as you discuss in your
"viral" section makes sense - I just don't think it makes sense over a
network. Also having multiple paths to get modules from makes sense].

I suspect having standard Unix scripts to set up the file structure (for big
publishers running their own Apache installation) and other helpers might
make it not too much harder than maintaining the raw directory structure.

One other thing: Troy, do your plans for sword-modules.org (or whatever it
was) have any bearing on this? For example, if it were relatively easy for
a repository to be set up there by Joe Random or Small Publisher X and
sword-modules.org provided FTP, it might lessen the advantage of HTTP over
FTP.

Jon
Post by Troy A. Griffitts
There are a number of finely delineated topics here we should separate.
1) Data format for our "module installation source".
i.e, is it a list of zips, is it an existing install set, etc.
a) Installing our software -- all implementations will be different
because we have a variety of packages running on a variety of platforms.
b) Installing modules from a module installation source
c) Running our software
The final item (c) seems out of place but really is important. We have
successfully achieved a common data storage format for an installation
set of books (i.e. "library/mods.d/..., library/modules/...) across a
vast number of platforms. Sure this hierarchy of files moves around to
different locations on different platforms, but once the location is
pinpointed, everything under that location remains the same for any of
our platforms.
The 'holy grail' for implementation of the (1) above is for both (b) and
(c) use cases to be handled by the SAME dataset, and is what we are
working towards. In other words, if an installed module library can,
itself, also be an installation source for other instances of any SWORD
software, then distribution of Bibles becomes viral. This brings us to
topic 2.
2) The viral ability END USERS to share modules.
If I can, for example,
o use the builtin installer of my SWORD software on my Android device
to
install my important modules
o then plug it into my friends computer and install modules FROM my
android device WITHOUT ANY PREPARATION
o then from his computer right-click on the top module library folder
and "Share as... 'SWORD Modules'",
o then go to any computer on the network and install modules from
that
"SWORD Modules" share,
o then install FileZilla on one of those computers and let anyone
over
the Internet install Bibles from that location,
o then point IIS or Apache to that location (http transport needs
work
but is on the way-- more talk below)
... distribution of Bibles then becomes viral. It is spontaneous. No
preparation. I have my library of books I use regularly with me and if
someone wants one of my books, then they can install straight from my
library.
3) The TRANSPORT TECHNOLOGY used to enable this over the Internet.
We chose FTP because it was more practical. It is very difficult to
implement the programmatic traversal of a directory and file hierarchy
over HTTP. Apache and IIS all spew out different html for directory
listings.
Now having said this, Nic Carter submitted the first pass at this which
works for our apache server (and likely many apache servers which spew
http://crosswire.org/ftpmirror/pub/
So, HTTP is currently possible in the engine and if we get push back
from publishers or find users not able to use FTP due to firewall
restrictions, we plan to fully support HTTP moving forward. We simply
need to break out our directory parsing code and make it work well with
a few of the more mainstream httpd services like Apache and IIS.
4) A single zip file for installation of a module (or modules) is simply
a zip of (1c) above.
Of the 3 'zip' formats we support for download from our server, only the
'rawzip' format should survive. I believe I can safely say we all agree
on that. This is in essence still the same thing. It is a module
repository of exactly 1 module in a difference storage medium. Heck it
could be 5 installed modules all zipped up and it would still be the
same thing. As I see it, it is an end user preference left to the
exercise of frontend developers. "Can my users operate their file
browser and drag and drop to my app easier than they can navigate their
"Browse to folder" dialog box in "Install SWORD Modules from path..."
option in my installer? If so, then I should support drag and drop on
my frontend-- after all it is simply an unzip. But the user gets no
benefit of our full installer: not description, to versioning upgrade
information if they already have an instance of that module installed,
to category, no language, etc.
5) If we find we cannot achieve this 'no prep' installation utopia in
the real world and it is necessary to have an intermediate format.
Matthew Talbert had a great suggestion on IRC last night to simply add a
'prep for publishing' (or in his words simply "publish") option to the
InstallMgr interface which would create (1) "module installation source"
data format if indeed it technically must be different. Great idea if
we need to concede to 2 disparate data formats, but I am still hopeful
we can avoid this extra step for would-be module distributors-- end
users and publishers alike.
Hope this separates out thoughts a little better.
Troy
Post by Greg Hellings
Post by Peter von Kaehne
I think Troy, the concern is correct.
For the publisher with some decent IT muscle and budget a proper repo
must be better, but for the small town church with a website and a couple of
modules to share - zip and http is a must.
Post by Greg Hellings
Post by Peter von Kaehne
Having a multiplicity of methods of getting modules into the system
would certainly be easier.
Post by Greg Hellings
Post by Peter von Kaehne
1) keep current methods - it is best for huge numbers of modules and it
is probably also best for anyone with enough money to have a fixed ip and a
server, able to run anonymous ftp
Post by Greg Hellings
While I very strongly agree that FTP should not be held up as the best
solution for everyone (I have worked with and for several
organizations who flat out refused to ever allow FTP of any type to be
a part of their systems and who primarily ran IIS for their web
content), I'm not sure the argument from excessive cost is accurate.
Annual DNS leasing from my provider is less than $40/year (I want to
say it's only about $17, but I might be remembering incorrectly)
Dedicated VPS hosting with dedicated IP address, about 20 GB of drive
space and full root access to my system is $19.95/mo.
If $23 per month is excessive for anyone who really intends to
distribute modules, they can almost certainly find someone else in the
SWORD community who will be willing to host it for them at very little
to no cost. Additionally I can host any low-volume traffic I want
from home by registering with a place like dyndns.com and setting up a
server on any connection that is moderately reliable (my home). The
option is not viable for people in developing countries but is
certainly more than viable for any first world country I'm aware of.
--Greg
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/0c247871/attachment-0001.html>
Greg Hellings
2010-11-07 20:48:08 UTC
Permalink
A lot here depends on evaluation of pros and cons.? I personally support
HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
list of all the books and where to find them for at least some of the
1. Minimising the number of GETs issued (and potentially file size and
download speed): I think this is particularly important for loading the
initial list of modules, which should preferably be as fast as it possibly
can be.? It also makes a difference for image modules like Dore's Woodcuts
with lots of loose files (which I remember taking a very long time to
install with Install Manager).
I'm not sure what your metrics are, but Ubuntu's apt for me hits about
70+ files on a half dozen or more servers to update its cache and this
takes less than 2 seconds. It takes installmgr almost no time to
process a single refresh.

The main problem is that we are not setup to handle display from
multiple sources and also not to refresh from all the repositories at
once, they have to be refreshed one at a time. This isn't so ideal to
the user's experience, but that's how it is done. I really doubt we
need to worry about the performance of hitting 30 or 40 repositories
if that is what the user selects to enable - and if we do, then that
indicates we need to rethink how we're hitting the repositories,
because the limit certainly isn't the number we'd be hitting at that
point.

--Greg
Jonathan Morgan
2010-11-08 11:06:25 UTC
Permalink
Post by Jonathan Morgan
Post by Jonathan Morgan
A lot here depends on evaluation of pros and cons. I personally support
HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
list of all the books and where to find them for at least some of the
1. Minimising the number of GETs issued (and potentially file size and
download speed): I think this is particularly important for loading the
initial list of modules, which should preferably be as fast as it
possibly
Post by Jonathan Morgan
can be. It also makes a difference for image modules like Dore's
Woodcuts
Post by Jonathan Morgan
with lots of loose files (which I remember taking a very long time to
install with Install Manager).
I'm not sure what your metrics are, but Ubuntu's apt for me hits about
70+ files on a half dozen or more servers to update its cache and this
takes less than 2 seconds. It takes installmgr almost no time to
process a single refresh.
Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds
to refresh the main CrossWire repository. My enduring memory of
InstallManager is that it feels slow. My guess is that the difference is
due to going from Oz to US rather than from US to US (I do have a reasonably
fast network connection). Apt on Ubuntu is a lot faster, but that is
because it is mirrored to an Oz site and doesn't have the latency.

I'm assuming that we have among our target audience:
1. People who are on the other side of the world from the US.
2. People who do not have a fast network connection.

The other thing is that I believe the Install Manager case we are testing *
is* in fact the case I advocate: It is grabbing mods.d.tar.gz rather than
doing FTP file traversal and grabbing one conf file at a time. I don't know
how to test it without rebuilding SWORD, but I think the InstallMgr would
grab one conf file at a time and so have a lot of latency from places like
here. This (a repository not requiring mods.d.tar.gz) is the simple FTP
server setup that is being promoted.
Post by Jonathan Morgan
The main problem is that we are not setup to handle display from
multiple sources and also not to refresh from all the repositories at
once, they have to be refreshed one at a time. This isn't so ideal to
the user's experience, but that's how it is done. I really doubt we
need to worry about the performance of hitting 30 or 40 repositories
if that is what the user selects to enable - and if we do, then that
indicates we need to rethink how we're hitting the repositories,
because the limit certainly isn't the number we'd be hitting at that
point.
If the vision is having individual publishers publishing their own work,
then having 30 or 40 repositories would be a big success - and I don't think
we can scale to that point (I don't see any prospect of that any time soon,
but if that is the vision it's worth thinking about whether we can support
it).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/1b78eac9/attachment-0001.html>
Greg Hellings
2010-11-10 02:50:33 UTC
Permalink
On Mon, Nov 8, 2010 at 7:48 AM, Greg Hellings <greg.hellings at gmail.com>
Post by Greg Hellings
A lot here depends on evaluation of pros and cons.? I personally support
HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
list of all the books and where to find them for at least some of the
1. Minimising the number of GETs issued (and potentially file size and
download speed): I think this is particularly important for loading the
initial list of modules, which should preferably be as fast as it possibly
can be.? It also makes a difference for image modules like Dore's Woodcuts
with lots of loose files (which I remember taking a very long time to
install with Install Manager).
I'm not sure what your metrics are, but Ubuntu's apt for me hits about
70+ files on a half dozen or more servers to update its cache and this
takes less than 2 seconds. ?It takes installmgr almost no time to
process a single refresh.
Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds
to refresh the main CrossWire repository.? My enduring memory of
InstallManager is that it feels slow.? My guess is that the difference is
due to going from Oz to US rather than from US to US (I do have a reasonably
fast network connection).? Apt on Ubuntu is a lot faster, but that is
because it is mirrored to an Oz site and doesn't have the latency.
On longer hauls over today's speeds, physical limitations of the speed
of light dominates for these small files. Even at only gigabit speeds
within the USA this can happen. Transocean it can be very prominent.
1. People who are on the other side of the world from the US.
2. People who do not have a fast network connection.
The other thing is that I believe the Install Manager case we are testing is
in fact the case I advocate: It is grabbing mods.d.tar.gz rather than doing
FTP file traversal and grabbing one conf file at a time.? I don't know how
to test it without rebuilding SWORD, but I think the InstallMgr would grab
one conf file at a time and so have a lot of latency from places like here.
This (a repository not requiring mods.d.tar.gz) is the simple FTP server
setup that is being promoted.
This is basically the model that Ubuntu repositories use. Every
repository has a single giant file with meta info on all the available
files. However, Debian also ships with a package (debmirror) which
makes it almost trivial to setup a local mirror, it will create the
tar file, etc. Any brief little Perl script could do this, which is
what Debmirror uses, or even a Python script which could be run raw as
Python on Linux and even compiled into a binary for our Windows
friends. Then the barrier to entry is no bigger than saying "Point an
FTP or HTTP server at your directory, set this little utility to run
daily, and you're good to go."

--Greg

Jonathan Morgan
2010-11-05 13:26:31 UTC
Permalink
Post by Peter von Kaehne
I think Troy, the concern is correct.
For the publisher with some decent IT muscle and budget a proper repo must
be better, but for the small town church with a website and a couple of
modules to share - zip and http is a must.
Having a multiplicity of methods of getting modules into the system would
certainly be easier.
1) keep current methods - it is best for huge numbers of modules and it is
probably also best for anyone with enough money to have a fixed ip and a
server, able to run anonymous ftp
2) add methods for local installation of zips. Look at MK Bible - pull a
zip over the programme and it gets installed. This is - emphatically - not
how I would want to install a large selection of modules or how I would want
to publish the same, but it is probably the best usability i have seen for a
frontend using only 2-3 modules (which is what MKBible is laid out for) and
a small time publisher or someone who has target audience of little computer
literacy
It's capable of far more than just dealing with small numbers of books when
you have two tools:
1. DownThemAll!: Download multiple zips at one go.
2. Multi-selection in Explorer / file window: I tried this in MK, and it
didn't seem to work correctly, but I probably don't have the latest
version. It definitely works in BPBible, and allows you to install multiple
books in one go.

(though I'm not convinced that a large percentage has these tools at their
disposable or is aware of them).

While drag and drop installation has a certain coolness factor, I feel
having a menu option (like the "File > Install Books" BPBible has, also
including multiple book installation) is more discoverable and thus perhaps
more useful to the starting off user. Also, people coming from the
background of e-Sword or similar tools are probably used to seeing a large
collection of books on a web page to download, and when they see a similar
list at Crosswire they do the same: download books and look for a way to
install them, while some people just like downloading a thing to make sure
they have it and could it share it with others if they wanted to (though
with "the cloud" this is probably less common than it used to be).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101106/c923bea0/attachment.html>
Peter von Kaehne
2010-11-05 13:34:10 UTC
Permalink
Von: Jonathan Morgan <jonmmorgan at gmail.com>
(though I'm not convinced that a large percentage has these tools at their
disposable or is aware of them).
At which point that particular debate probably ends :-)
While drag and drop installation has a certain coolness factor, I feel
having a menu option (like the "File > Install Books" BPBible has, also
including multiple book installation) is more discoverable and thus perhaps
more useful to the starting off user.
No objection, but also no contradiction.
Also, people coming from the
background of e-Sword or similar tools are probably used to seeing a large
collection of books on a web page to download, and when they see a similar
list at Crosswire they do the same: download books and look for a way to
install them, while some people just like downloading a thing to make sure
they have it and could it share it with others if they wanted to (though
with "the cloud" this is probably less common than it used to be).
I think the huge number of support emails "I have downloaded x number of modules and now what am I supposed to do" suggests the same - though of course us taking away the zips from the webpage would be a helpful step to stop that.

Downloading a zip for sharing is of course a very useful way to pass about modules via sneaker net. And that is in turn a way of some importance where the interent is either sparse or controlled. I think we acknowledge this by making the zip's available but we do not exactly facilitate it beyond that point. And that is a shame.

Peter
Jon
--
GRATIS! Movie-FLAT mit ?ber 300 Videos.
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome
DM Smith
2010-11-05 14:00:35 UTC
Permalink
Post by Peter von Kaehne
Von: Jonathan Morgan<jonmmorgan at gmail.com>
(though I'm not convinced that a large percentage has these tools at their
disposable or is aware of them).
At which point that particular debate probably ends :-)
While drag and drop installation has a certain coolness factor, I feel
having a menu option (like the "File> Install Books" BPBible has, also
including multiple book installation) is more discoverable and thus perhaps
more useful to the starting off user.
No objection, but also no contradiction.
Also, people coming from the
background of e-Sword or similar tools are probably used to seeing a large
collection of books on a web page to download, and when they see a similar
list at Crosswire they do the same: download books and look for a way to
install them, while some people just like downloading a thing to make sure
they have it and could it share it with others if they wanted to (though
with "the cloud" this is probably less common than it used to be).
I think the huge number of support emails "I have downloaded x number of modules and now what am I supposed to do" suggests the same - though of course us taking away the zips from the webpage would be a helpful step to stop that.
We should at least drop the MacOSX ones. And perhaps the Windows ones.

If someone has installed Xiphos or Bible Desktop, but never has
installed BibleCS, then these don't work. They also don't work on 64-bit
(according to reports).

IIRC: Troy said these were there to satisfy a publisher's request that
some info from a module's conf be shown for modules installed this way.
It only ever worked in BibleCS. Is that still a valid concern?
Post by Peter von Kaehne
Downloading a zip for sharing is of course a very useful way to pass about modules via sneaker net. And that is in turn a way of some importance where the interent is either sparse or controlled. I think we acknowledge this by making the zip's available but we do not exactly facilitate it beyond that point. And that is a shame.
I think every frontend should be able to browse, at least locally, for a
"raw" zip and install it. That way the downloads from the web page
become less of a support question.

On a different note, I think it might be good to have multiple install
locations, one for each repository. Today, we have a KJV, but if this
gets going I could see a publisher with their own repository having a
KJV. JSword *had* a mechanism to allow for duplicates such as these by
disambiguating them by their repository name, e.g.
KJV (CrossWire)
We pulled it because it didn't happen in reality and because it was not
implemented very well.

In this situation, zips are anonymous and would go in a "private"
repository.

Today, JSword allow for multiple repository locations, but only one for
install. I use my own private repository for modules I have created so
that I don't accidentally wack them.

DM
Post by Peter von Kaehne
Peter
Jon
Troy A. Griffitts
2010-11-05 14:49:58 UTC
Permalink
Post by DM Smith
IIRC: Troy said these were there to satisfy a publisher's request that
some info from a module's conf be shown for modules installed this way.
It only ever worked in BibleCS. Is that still a valid concern?
Well, almost.

There is a feature in the engine where you can specify in your module
repository:

[Globals]
AutoInstall=newmods

where newmods is a folder which lives at the same place as mods.d/ and
modules/

on loading of a module repository, SWMgr does a scan of this folder for
.conf files and will copy them to the mods.d folder, but first will call
a hook method SWMgr::AddModToConfig which allows a fontend to display a
nice dialog box:

"Hey, you've installed a new module! Here's the About information..."

or whatever.

This mechanism was, as DM mentions, added at the request of a publisher
when BibleCS was the only frontend available. Permission to use their
module was predicated on us showing their about information upon
installation of their module.

I'd definitely like to remove the Win32 specific InstallShield module
install packages.

BibleCS InstallMgr is undergoing some "play nice with other frontends"
updates right now looking toward an eminent update of BibleCS compiled
against 1.6.2 and with av11n support. I suggest after this release we
remove the windows installer files.

Do we have any feedback on the MacOSX installer files?

Troy
Jonathan Morgan
2010-11-05 15:01:17 UTC
Permalink
Post by Troy A. Griffitts
Post by DM Smith
IIRC: Troy said these were there to satisfy a publisher's request that
some info from a module's conf be shown for modules installed this way.
It only ever worked in BibleCS. Is that still a valid concern?
Well, almost.
There is a feature in the engine where you can specify in your module
[Globals]
AutoInstall=newmods
where newmods is a folder which lives at the same place as mods.d/ and
modules/
on loading of a module repository, SWMgr does a scan of this folder for
.conf files and will copy them to the mods.d folder, but first will call
a hook method SWMgr::AddModToConfig which allows a fontend to display a
"Hey, you've installed a new module! Here's the About information..."
or whatever.
This mechanism was, as DM mentions, added at the request of a publisher
when BibleCS was the only frontend available. Permission to use their
module was predicated on us showing their about information upon
installation of their module.
I'd definitely like to remove the Win32 specific InstallShield module
install packages.
BibleCS InstallMgr is undergoing some "play nice with other frontends"
updates right now looking toward an eminent update of BibleCS compiled
against 1.6.2 and with av11n support. I suggest after this release we
remove the windows installer files.
Do we have any feedback on the MacOSX installer files?
This was discussed extensively over a year ago. See
http://www.mail-archive.com/sword-devel at crosswire.org/msg18634.html for
Manfred's response to that thread. The relevant quotations would be:
"Mac OS X zip file is still supported and we will keep support.

Actually it only is a zipped folder that has a ".swd" extension which is
registered for MacSword in the OS and is shown as a file in the OS file
manager. This makes it possible that you can have MacSword openened
automatically when the file is opened. But we also support just placing the
unzipped raw folder in the Mac OS X default module location path."

Given that response, I'm not sure there is enough advantage in providing two
separate options (if the Window zip goes) just to have a separate file
extension for Mac (though in some ways I think having a separate file
extension would be nice, so that people treat it as a distinct entity rather
than just a zip file that gets confused with the hundreds of other zip files
that they have in their Download directory - or at least that I do).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101106/c94f8d1e/attachment.html>
DM Smith
2010-11-05 15:41:45 UTC
Permalink
Post by Troy A. Griffitts
Do we have any feedback on the MacOSX installer files?
I think the download pages should clearly say, perhaps on every page to
use the program's built-in installer.

We get lots of support questions from MacSword and BibleDesktop users on
how to install the downloads from CrossWire on a Mac. The answer always
is the same: Use your program's install manager.

The Mac downloads only work for MacSword. Unzip on a Mac creates a
folder with the same name as the file, but without the zip extension. By
having KJV.swd.zip, this produces KJV.swd on the desktop (or in the
download folder or where ever). If MacSword is installed then the *.swd
folders don't look like a folder, but a MacOS package and the contents
are hidden in Finder. One can double click on it to open it in MacSword.
But unless the user moves it to a folder that MacSword looks at (IIRC:
~/Library/Application Support/Sword), it won't be seen by the program.
It is not very Mac like to have people drag stuff into
~/Library/Application Support.

So we haven't been explaining that to them. (I field MacSword questions
to CrossWire's support from time-to-time.)

Also,I don't think that PocketSword can use them.

I defer to Manfred on whether to keep them. I thought that he said here
that they could go away now that MacSword has an installer. But I can't
find it now.

In Him,
DM
Manfred Bergmann
2010-11-05 22:01:15 UTC
Permalink
Post by Troy A. Griffitts
Do we have any feedback on the MacOSX installer files?
I think the download pages should clearly say, perhaps on every page to use the program's built-in installer.
We get lots of support questions from MacSword and BibleDesktop users on how to install the downloads from CrossWire on a Mac. The answer always is the same: Use your program's install manager.
The Mac downloads only work for MacSword. Unzip on a Mac creates a folder with the same name as the file, but without the zip extension. By having KJV.swd.zip, this produces KJV.swd on the desktop (or in the download folder or where ever). If MacSword is installed then the *.swd folders don't look like a folder, but a MacOS package and the contents are hidden in Finder. One can double click on it to open it in MacSword. But unless the user moves it to a folder that MacSword looks at (IIRC: ~/Library/Application Support/Sword), it won't be seen by the program. It is not very Mac like to have people drag stuff into ~/Library/Application Support.
So we haven't been explaining that to them. (I field MacSword questions to CrossWire's support from time-to-time.)
Also,I don't think that PocketSword can use them.
I defer to Manfred on whether to keep them. I thought that he said here that they could go away now that MacSword has an installer. But I can't find it now.
Well. Actually I would keep them because they are quite comfortable to use on a Mac because of how they are treated by Finder and MacSword.
But I understand that it is confusing to have three different module download options on our web page.
The user can, if he wishes, rename the unzipped zip download and add the ".swd".
And since we have the install manager I guess most people do use it.



Manfred
Nic Carter
2010-11-06 04:19:19 UTC
Permalink
Post by Troy A. Griffitts
Do we have any feedback on the MacOSX installer files?
I think the download pages should clearly say, perhaps on every page to use the program's built-in installer.
We get lots of support questions from MacSword and BibleDesktop users on how to install the downloads from CrossWire on a Mac. The answer always is the same: Use your program's install manager.
...
Also,I don't think that PocketSword can use them.
No, PocketSword doesn't use them. PocketSword has support for installing a "raw zip" via a web browser (simple http form that can submit a zip file to the device) (which can, incidentally, contain several modules zipped together) and use of the InstallManager.

My suggestion (as a Mac user and MacSword user) would be to simplify things by only having "raw zip" modules available for download, and push people to use the InstallManager. Perhaps even add an annoying page to the jsp download pages such that if a user clicks on the download link, it initially displays an (annoying!) page describing that the user should be using the internal InstallManager functionality of their frontend, and then providing a link to "actually" download the module. :) That way we can hopefully get users to install modules the "easy" way, rather than "manually". :) Yes, it would be annoying, but hopefully the annoyingness of it all would mean users would be persuaded to do things the way we'd like? ;)

Just my random thoughts... :)
Jonathan Morgan
2010-11-05 14:02:26 UTC
Permalink
Post by Peter von Kaehne
Von: Jonathan Morgan <jonmmorgan at gmail.com>
(though I'm not convinced that a large percentage has these tools at
their
disposable or is aware of them).
At which point that particular debate probably ends :-)
Indeed :).
Post by Peter von Kaehne
While drag and drop installation has a certain coolness factor, I feel
having a menu option (like the "File > Install Books" BPBible has, also
including multiple book installation) is more discoverable and thus perhaps
more useful to the starting off user.
No objection, but also no contradiction.
Both things (I suspect) require frontend support as well as SWORD support,
if any. I suspect drag and drop needs a little more work to implement than
a menu option popping up a file chooser, and then delegating to SWORD, but
I'm really not sure.
Post by Peter von Kaehne
Also, people coming from the
background of e-Sword or similar tools are probably used to seeing a
large
collection of books on a web page to download, and when they see a
similar
list at Crosswire they do the same: download books and look for a way to
install them, while some people just like downloading a thing to make
sure
they have it and could it share it with others if they wanted to (though
with "the cloud" this is probably less common than it used to be).
I think the huge number of support emails "I have downloaded x number of
modules and now what am I supposed to do" suggests the same - though of
course us taking away the zips from the webpage would be a helpful step to
stop that.
Downloading a zip for sharing is of course a very useful way to pass about
modules via sneaker net. And that is in turn a way of some importance where
the interent is either sparse or controlled. I think we acknowledge this by
making the zip's available but we do not exactly facilitate it beyond that
point. And that is a shame.
Two points here:
1. Part of the trouble comes from having multiple download options (e.g. Raw
Zip, Mac and Windows). I have in the past suggested that there only be one
(and that that one be the Raw Zip, as the most universal). Apparently
BPBible supports all three download formats, but I can't expect everyone
else to do that, and it's still potentially quite confusing.

2. Offering a list of downloads at CrossWire tends to suggest that they are
the *only* books available. While I can download zip files from Xiphos FTP
directly (for example), it's not easy to find out about, and I'm sure some
people evaluate the software purely on the basis of the books available
without even trying the software.

I think the latter point is probably the more important of the two. While
it is no doubt a worthy goal to have publishers independently able to
publish their own work, I expect to be able to come to a software
organisation's website and see what books are available from that website
for that software. I don't know if we can facilitate this discovery from
the website (and if so how) but people do judge the software on the library
and so this is something that bears thinking about. (Once we get to the
Logoses of this world people also expect an integrated web store which
allows you to purchase books and keeps track of every book you have access
to and allows you to sync between software, etc. I don't think we're going
there any time soon...)

BPBible relies on zip installation at present. I think this is a reasonable
decision, though we would like to grow Install Manager support at some point
(in part to get easier access to the Xiphos repository).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101106/53f5a1af/attachment-0001.html>
Karl Kleinpaste
2010-11-05 14:24:30 UTC
Permalink
Post by Jonathan Morgan
2. Offering a list of downloads at CrossWire tends to suggest that
they are the *only* books available. While I can download zip files
from Xiphos FTP directly (for example), it's not easy to find out
about
...
Post by Jonathan Morgan
BPBible relies on zip installation at present. I think this is a
reasonable decision, though we would like to grow Install Manager
support at some point (in part to get easier access to the Xiphos
repository).
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open,
and see what's available. Just two assumptions:
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Post by Jonathan Morgan
I think every frontend should be able to browse, at least locally, for
a "raw" zip and install it. That way the downloads from the web page
become less of a support question.
You are re-invoking a question I asked some months ago, about baseline
behavior -vs- per-app features. For all apps to support such behavior
is to draw a rather hard line in the sand under the "baseline" category.
Jonathan Morgan
2010-11-05 14:35:01 UTC
Permalink
Post by Karl Kleinpaste
Post by Jonathan Morgan
2. Offering a list of downloads at CrossWire tends to suggest that
they are the *only* books available. While I can download zip files
from Xiphos FTP directly (for example), it's not easy to find out
about
...
Post by Jonathan Morgan
BPBible relies on zip installation at present. I think this is a
reasonable decision, though we would like to grow Install Manager
support at some point (in part to get easier access to the Xiphos
repository).
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open,
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Agreed. I can and have. I don't expect my average user to, though.
Post by Karl Kleinpaste
Post by Jonathan Morgan
I think every frontend should be able to browse, at least locally, for
a "raw" zip and install it. That way the downloads from the web page
become less of a support question.
You are re-invoking a question I asked some months ago, about baseline
behavior -vs- per-app features. For all apps to support such behavior
is to draw a rather hard line in the sand under the "baseline" category.
I think it should be baseline for the same reasons as DM. However, since we
have very little that is truly "baseline" this would probably just join the
list of "would be nice to have, but too much work for right now" (and I
willingly concede that BPBible has this too).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101106/43ab3b23/attachment.html>
Karl Kleinpaste
2010-11-05 14:53:01 UTC
Permalink
Post by Jonathan Morgan
Post by Karl Kleinpaste
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open,
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Agreed. I can and have. I don't expect my average user to, though.
My point was that a module manager (that is, the InstallManager class)
can do this readily. It should be invisible for http+zip retrieval,
just as it already is for ftp+components retrieval. It should just
happen as part of the request for local refresh of the remote content.

The downside is that, for ftp access today, mods.d.tar.gz is technically
not required; in its absence, InstallManager will crawl into mods.d and
find *.conf the hard way. I don't get warm, fuzzy feelings about the
possibility of implementing an adequately general and successful parsing
scheme of directory content via http access.
Jonathan Morgan
2010-11-05 23:24:19 UTC
Permalink
Post by Karl Kleinpaste
Post by Jonathan Morgan
Post by Karl Kleinpaste
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it
open,
Post by Jonathan Morgan
Post by Karl Kleinpaste
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Agreed. I can and have. I don't expect my average user to, though.
My point was that a module manager (that is, the InstallManager class)
can do this readily. It should be invisible for http+zip retrieval,
just as it already is for ftp+components retrieval. It should just
happen as part of the request for local refresh of the remote content
Indeed. But BPBible does not have InstallManager support.

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101106/3eda9d01/attachment-0001.html>
Nic Carter
2010-11-06 04:36:51 UTC
Permalink
Post by Karl Kleinpaste
Post by Jonathan Morgan
Post by Karl Kleinpaste
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open,
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Agreed. I can and have. I don't expect my average user to, though.
My point was that a module manager (that is, the InstallManager class)
can do this readily. It should be invisible for http+zip retrieval,
just as it already is for ftp+components retrieval. It should just
happen as part of the request for local refresh of the remote content.
The downside is that, for ftp access today, mods.d.tar.gz is technically
not required; in its absence, InstallManager will crawl into mods.d and
find *.conf the hard way. I don't get warm, fuzzy feelings about the
possibility of implementing an adequately general and successful parsing
scheme of directory content via http access.
Actually, I was planning on implementing this for PocketSword in the near future. The idea was that I was going to move completely away from FTP and only support HTTP. But given that parsing HTTP is too painful (I initially submitted a patch for HTTP parsing, but it only works for CrossWire and not for the Bible.org nor for the Xiphos repos, and I have no intention of modifying the parsing code even more in order to try to support more web servers!), I was going to download the mods.d.tar.gz file (over HTTP) and use that for module information. Then download the ZIP of the module over HTTP. If a repo doesn't have the zip folder (or the mods.d.tar.gz file), then I would either have a fall-back to use the normal InstallManager, or perhaps I'd just say that PocketSword wouldn't support that repo. But after a year of PocketSword doing things the "right" way, I've decided to take the JSword approach and hope that doesn't cause too many issues... :(

FYI, my personal tests on an iOS device show that accessing the CrossWire repo via FTP is reasonably slower than accessing it via HTTP. And that's simply by using the existing InstallManager code (including my HTTP parsing patch) and switching protocols between FTP & HTTP. And given that PocketSword only runs on iOS devices, I want the reasonably large performance gains that I can gain from simply switching protocols! :)

Given my lack of free programming hours atm, this isn't going to happen this year. But it is in the back of my head to implement at some stage in the future. I also have some other "urgent" fixes to do to PocketSword's downloads mechanism, such as providing background downloading. I mentioned to David the other day that I'm finishing up my current job at the end of the year in order to resume studying next year, but will have a few weeks of (hopefully!) full-time work on PocketSword in order to try to catch up on some of these "urgent" tasks.

BUT
I'm glad this discussion is taking place now, so I can see the resolution of all of this, and that may influence my decision on how I proceed with "improving" PocketSword's download mechanism.


Thanks heaps for all the discussion so far, please keep it coming!
ybic
nic... :)
Troy A. Griffitts
2010-11-06 11:15:22 UTC
Permalink
Post by Nic Carter
I initially submitted a patch for HTTP parsing, but it only
works for CrossWire and not for the Bible.org nor for the Xiphos
repos, and I have no intention of modifying the parsing code even
more in order to try to support more web servers!
:) thanks for the patch! Yeah, surprisingly even FTP directory parsing
is painful. Even libCURL doesn't have an FTP directory listing parse
function. I couldn't believe that when I wrote the FTP code! We found
a portable library call ftpparse which parses directory listings for us.
When doing the HTTP transport, I was hopeful we might find an
httpdirparse or something :) But no such luck, as of yet.

We were talking on #sword the other day about how odd it is that there
is no w3c standard for the obvious use case:

Browse a hierarchy of folders+resources and retrieve some.

I brought this up with a frequent member of w3c committees and he
suggested we develop a silly stupid minimal schema to represent a
resource tree and a) submit it for w3c approval, and b) submit updates
for Apache and IIS to update their Folder Index listings to comply to
the proposal. e.g., something like,

<?stylesheet href="apache_look_and_feel.css"?>
<folder name="My Documents">
<resource
type="file" mimetype="application/msword" name="War Of the Worlds.doc"/>
</folder>

Then, end users wouldn't notice a difference, and we could have a
standard to easily parse. As always, you know who is always in the
details: attributes for permissions, mtime...; do you make the whole
subdirectory hierarchy available from a directory request, or just the
immediate children...

Anyway, we can dream of a bold new Internet where everything is
standardized and straightforward for developers... :) ahhhhh.


Troy
Nic Carter
2010-11-07 02:22:36 UTC
Permalink
Thanks for the long reply. :) It is frustrating that some things are designed really well and other bits are painful, like getting a listing of files...

ok, so, I did take a quick look at my code and one way of fixing the Xiphos listing, for example, is by not trying to parse the file sizes. We could trust the module size provided in the module conf (which isn't always provided and is sometimes wrong!) and then simply have a total size indicator rather than have an indicator for each file downloaded in the module. I think that may also work for bible.org? Would be an option, and then we'd need to check to see if the file size returned by FTPTransport::getDirList() is zero, and if it is, use the total size provided by the module conf. But, I believe there is a way of telling the size of a file being retrieved via HTTP GET? hopefully we could use that as well? :)

Anyway, just a quick thought. Of course, I'd prefer that we didn't try to parse a file listing like this . . . and would rather we used the mods.d.tar.gz file & module ZIP files, but more on that in another email. :)


Thanks, ybic
nic... :)

ps: I wouldn't be against someone submitting something to w3c, but I wouldn't be holding my breath for it to be implemented, let alone approved... :( :(
Post by Troy A. Griffitts
Post by Nic Carter
I initially submitted a patch for HTTP parsing, but it only
works for CrossWire and not for the Bible.org nor for the Xiphos
repos, and I have no intention of modifying the parsing code even
more in order to try to support more web servers!
:) thanks for the patch! Yeah, surprisingly even FTP directory parsing
is painful. Even libCURL doesn't have an FTP directory listing parse
function. I couldn't believe that when I wrote the FTP code! We found
a portable library call ftpparse which parses directory listings for us.
When doing the HTTP transport, I was hopeful we might find an
httpdirparse or something :) But no such luck, as of yet.
We were talking on #sword the other day about how odd it is that there
Browse a hierarchy of folders+resources and retrieve some.
I brought this up with a frequent member of w3c committees and he
suggested we develop a silly stupid minimal schema to represent a
resource tree and a) submit it for w3c approval, and b) submit updates
for Apache and IIS to update their Folder Index listings to comply to
the proposal. e.g., something like,
<?stylesheet href="apache_look_and_feel.css"?>
<folder name="My Documents">
<resource
type="file" mimetype="application/msword" name="War Of the Worlds.doc"/>
</folder>
Then, end users wouldn't notice a difference, and we could have a
standard to easily parse. As always, you know who is always in the
details: attributes for permissions, mtime...; do you make the whole
subdirectory hierarchy available from a directory request, or just the
immediate children...
Anyway, we can dream of a bold new Internet where everything is
standardized and straightforward for developers... :) ahhhhh.
Troy
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Weston Ruter
2010-11-07 10:34:49 UTC
Permalink
Actually, directory indexes are standardized in WebDAV. If the problem with
using HTTP is that the directory listings aren't provided in a standardized
format across Apache and IIS but rather are in a sort of ad hoc HTML format,
a solution could be to provide a custom directory index handler that returns
a WebDAV collection (with depth = 1) which is in the WebDAV XML format.
Normally (according <http://en.wikipedia.org/wiki/WebDAV> to Wikipedia),
WebDAV specifies you use a
PROPFIND<http://msdn.microsoft.com/en-us/library/aa142960%28EXCHG.65%29.aspx>
WebDAV HTTP method when requesting a directory to "retrieve the collection
structure (a.k.a. directory hierarchy) of a remote system", but there's no
reason why you can't also return this same XML content type when performing
a GET request of a directory.

For example, issuing a PROPFIND request of a WordPress SVN tag, which when
making a GET request returns the regular Apache directory listing:

curl -i -H "Depth: 1" -X PROPFIND
http://svn.automattic.com/wordpress/tags/3.0.1

Here's the output: https://gist.github.com/05c389c81ccc8fc2f20b

This same standard WebDAV format could be the response of a regular GET
request. In Apache, this could be accomplished by writing an mod_rewrite
rule which matches all directory requests and directs them to a PHP script
which returns the directory listing in the WebDAV XML format, for example:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . /path/to/directory-list.php [L]
</IfModule>

Just a thought,
Weston
Post by Nic Carter
Thanks for the long reply. :) It is frustrating that some things are
designed really well and other bits are painful, like getting a listing of
files...
ok, so, I did take a quick look at my code and one way of fixing the Xiphos
listing, for example, is by not trying to parse the file sizes. We could
trust the module size provided in the module conf (which isn't always
provided and is sometimes wrong!) and then simply have a total size
indicator rather than have an indicator for each file downloaded in the
module. I think that may also work for bible.org? Would be an option,
and then we'd need to check to see if the file size returned by
FTPTransport::getDirList() is zero, and if it is, use the total size
provided by the module conf. But, I believe there is a way of telling the
size of a file being retrieved via HTTP GET? hopefully we could use that as
well? :)
Anyway, just a quick thought. Of course, I'd prefer that we didn't try to
parse a file listing like this . . . and would rather we used the
mods.d.tar.gz file & module ZIP files, but more on that in another email.
:)
Thanks, ybic
nic... :)
ps: I wouldn't be against someone submitting something to w3c, but I
wouldn't be holding my breath for it to be implemented, let alone
approved... :( :(
Post by Troy A. Griffitts
Post by Nic Carter
I initially submitted a patch for HTTP parsing, but it only
works for CrossWire and not for the Bible.org nor for the Xiphos
repos, and I have no intention of modifying the parsing code even
more in order to try to support more web servers!
:) thanks for the patch! Yeah, surprisingly even FTP directory parsing
is painful. Even libCURL doesn't have an FTP directory listing parse
function. I couldn't believe that when I wrote the FTP code! We found
a portable library call ftpparse which parses directory listings for us.
When doing the HTTP transport, I was hopeful we might find an
httpdirparse or something :) But no such luck, as of yet.
We were talking on #sword the other day about how odd it is that there
Browse a hierarchy of folders+resources and retrieve some.
I brought this up with a frequent member of w3c committees and he
suggested we develop a silly stupid minimal schema to represent a
resource tree and a) submit it for w3c approval, and b) submit updates
for Apache and IIS to update their Folder Index listings to comply to
the proposal. e.g., something like,
<?stylesheet href="apache_look_and_feel.css"?>
<folder name="My Documents">
<resource
type="file" mimetype="application/msword" name="War Of the
Worlds.doc"/>
Post by Troy A. Griffitts
</folder>
Then, end users wouldn't notice a difference, and we could have a
standard to easily parse. As always, you know who is always in the
details: attributes for permissions, mtime...; do you make the whole
subdirectory hierarchy available from a directory request, or just the
immediate children...
Anyway, we can dream of a bold new Internet where everything is
standardized and straightforward for developers... :) ahhhhh.
Troy
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
Weston Ruter
http://weston.ruter.net/
@westonruter <http://twitter.com/westonruter> - Google
Profile<http://www.google.com/profiles/WestonRuter#about>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101107/272e8f0e/attachment.html>
DM Smith
2010-11-07 12:11:51 UTC
Permalink
Post by Nic Carter
But, I believe there is a way of telling the size of a file being retrieved via HTTP GET? hopefully we could use that as well? :)
IIRC, it's HEAD. JSword uses it. Works well.
In Him,
DM

From my phone.
Weston Ruter
2010-11-07 13:35:52 UTC
Permalink
Yes, do a HEAD request and then look at the Content-Length header.
Post by Nic Carter
Post by Nic Carter
But, I believe there is a way of telling the size of a file being
retrieved via HTTP GET? hopefully we could use that as well? :)
IIRC, it's HEAD. JSword uses it. Works well.
In Him,
DM
From my phone.
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
Weston Ruter
http://weston.ruter.net/
@westonruter <http://twitter.com/westonruter> - Google
Profile<http://www.google.com/profiles/WestonRuter#about>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101107/62cc670e/attachment.html>
Jonathan Morgan
2010-11-08 11:38:14 UTC
Permalink
Just recall that HEAD presumably takes another roundtrip to the server,
adding latency, while InstallSize just requires you to have the conf files
you have probably already got. (In reality, it's probably not that much of
a problem, but I'm obviously feeling paranoid about latency today ;) ).

Jon
Post by Weston Ruter
Yes, do a HEAD request and then look at the Content-Length header.
Post by Nic Carter
Post by Nic Carter
But, I believe there is a way of telling the size of a file being
retrieved via HTTP GET? hopefully we could use that as well? :)
IIRC, it's HEAD. JSword uses it. Works well.
In Him,
DM
From my phone.
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
Weston Ruter
http://weston.ruter.net/
@westonruter <http://twitter.com/westonruter> - Google Profile<http://www.google.com/profiles/WestonRuter#about>
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/c3a6a41b/attachment.html>
DM Smith
2010-11-08 12:09:36 UTC
Permalink
InstallSize doesn't give the download size. HEAD does. For compressed modules and image modules these are fairly close. InstallSize is also fairly recent and not all confs have it. So if our software did use it, it'd have too fall back to another way when not there.


Doing HEAD on all the parts and summing it is a bother and might involve many latencies (eg mage modules) but is singular for a zip and thus no big deal.

In Him,
DM

Sent from my phone
Just recall that HEAD presumably takes another roundtrip to the server, adding latency, while InstallSize just requires you to have the conf files you have probably already got. (In reality, it's probably not that much of a problem, but I'm obviously feeling paranoid about latency today ;) ).
Jon
Yes, do a HEAD request and then look at the Content-Length header.
Post by Nic Carter
But, I believe there is a way of telling the size of a file being retrieved via HTTP GET? hopefully we could use that as well? :)
IIRC, it's HEAD. JSword uses it. Works well.
In Him,
DM
Post by Nic Carter
From my phone.
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
Weston Ruter
http://weston.ruter.net/
@westonruter - Google Profile
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/111c9050/attachment-0001.html>
Jonathan Morgan
2010-11-08 12:25:09 UTC
Permalink
Post by DM Smith
InstallSize doesn't give the download size. HEAD does. For compressed
modules and image modules these are fairly close. InstallSize is also fairly
recent and not all confs have it. So if our software did use it, it'd have
too fall back to another way when not there.
True indeed (though if it were in a standard script for managing a
repository, like Karl has it, or part of a standard module preparation
procedure it would probably mostly just work). However, I think we
are talking about different uses here. From memory, the use case for
InstallSize is to have something to give the user a rough idea about
the size before choosing to install it. This needs to be something in
the conf file so that it is only got once up front, along with
everything else in mods.d.tar.gz, rather than being got once per file.
When downloading an individual zip file you probably want HEAD to get
an exact size for that particular module.
Post by DM Smith
Doing HEAD on all the parts and summing it is a bother and might involve
many latencies (eg mage modules) but is singular for a zip and thus no big
deal.
Yes.

Jon
Karl Kleinpaste
2010-11-08 13:05:35 UTC
Permalink
Post by DM Smith
InstallSize is also fairly recent and not all confs have it.
All 3 CrossWire repos (regular, beta, experimental) and the Xiphos repo
have InstallSize support.

Whether other publishers put it to use is an unknown.

For CrossWire repos, it's a nightly cron-driven script. All *.conf gain
InstallSize if they do not already have it.
Butrus Damaskus
2010-11-08 21:17:19 UTC
Permalink
Post by Nic Carter
I initially submitted a patch for HTTP parsing, but it only
works for CrossWire and not for the Bible.org nor for the Xiphos
repos, and I have no intention of modifying the parsing code even
more in order to try to support more web servers!
:) thanks for the patch! ?Yeah, surprisingly even FTP directory parsing
is painful. ?Even libCURL doesn't have an FTP directory listing parse
function. ?I couldn't believe that when I wrote the FTP code! ?We found
a portable library call ftpparse which parses directory listings for us.
?When doing the HTTP transport, I was hopeful we might find an
httpdirparse or something :) ?But no such luck, as of yet.
We were talking on #sword the other day about how odd it is that there
Browse a hierarchy of folders+resources and retrieve some.
I brought this up with a frequent member of w3c committees and he
suggested we develop a silly stupid minimal schema to represent a
resource tree and a) submit it for w3c approval, and b) submit updates
for Apache and IIS to update their Folder Index listings to comply to
the proposal. ?e.g., something like,
<?stylesheet href="apache_look_and_feel.css"?>
<folder name="My Documents">
?<resource
? ?type="file" mimetype="application/msword" name="War Of the Worlds.doc"/>
</folder>
Hm, don't understand, why you want to reinvent the wheel, when WebDAV exists.
David Overcash
2010-11-08 21:21:16 UTC
Permalink
Post by Butrus Damaskus
Hm, don't understand, why you want to reinvent the wheel, when WebDAV exists.
To me it seems that it would make mobile development much easier. Then
again, I have yet to try with WebDAV.

It's just that WebDAV (in a sense) has already tried to reinvent the wheel -
and all that is necessary is simple HTTP.

Cheers,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/9e01a164/attachment.html>
Butrus Damaskus
2010-11-08 21:26:52 UTC
Permalink
On Mon, Nov 8, 2010 at 10:21 PM, David Overcash
Post by Butrus Damaskus
Hm, don't understand, why you want to reinvent the wheel, when WebDAV exists.
To me it seems that it would make mobile development much easier. ?Then
again, I have yet to try with WebDAV.
It's just that WebDAV (in a sense) has already tried to reinvent the wheel -
and all that is necessary is simple HTTP.
NO, it hasn't!

WebDAV is build ON TOP of HTTP.
Cheers,
David
Matthew Talbert
2010-11-05 21:34:23 UTC
Permalink
Post by Karl Kleinpaste
Post by Jonathan Morgan
2. Offering a list of downloads at CrossWire tends to suggest that
they are the *only* books available. ?While I can download zip files
from Xiphos FTP directly (for example), it's not easy to find out
about
...
Post by Jonathan Morgan
BPBible relies on zip installation at present. ?I think this is a
reasonable decision, though we would like to grow Install Manager
support at some point (in part to get easier access to the Xiphos
repository).
You can access http://ftp.xiphos.org/sword/mods.d.tar.gz, crack it open,
- the zip subdir is at the same level as mods.d
- xyz.conf has a corresponding xyz.zip.
Agreed.? I can and have.? I don't expect my average user to, though.
Perhaps this isn't any better, but you can see the entire list here:
http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very
descriptive names, but it slightly more user friendly than opening the
tar.gz.

Matthew
Karl Kleinpaste
2010-11-05 21:48:57 UTC
Permalink
Post by Matthew Talbert
http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very
descriptive names, but it slightly more user friendly than opening the
tar.gz.
Well, again, I don't want a _human_ cracking open mods.d.tar.gz; I want
the app's installer to do it, and display a nice set of available
modules with descriptions and sizes and whatnot. I think InstallManager
should be taught to do it all, just the way it does for FTP.
Troy A. Griffitts
2010-11-06 11:45:19 UTC
Permalink
Sure, of course I agree with Karl. If we decide on expanding (or
contracting) the format of an installation repository, the InstallMgr
class in the engine will operate against that new definition and
frontend developers already using InstallMgr shouldn't need to change
any code.
Post by Karl Kleinpaste
Post by Matthew Talbert
http://ftp.xiphos.org/sword/zip/ Of course, it doesn't use very
descriptive names, but it slightly more user friendly than opening the
tar.gz.
Well, again, I don't want a _human_ cracking open mods.d.tar.gz; I want
the app's installer to do it, and display a nice set of available
modules with descriptions and sizes and whatnot. I think InstallManager
should be taught to do it all, just the way it does for FTP.
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
Troy A. Griffitts
2010-11-05 15:06:42 UTC
Permalink
Post by Jonathan Morgan
2. Offering a list of downloads at CrossWire tends to suggest that they
are the *only* books available. While I can download zip files from
Xiphos FTP directly (for example), it's not easy to find out about, and
I'm sure some people evaluate the software purely on the basis of the
books available without even trying the software.
Since those pages are already generated dynamically, I suppose we could
simply use our own InstallMgr class to browse all known module
repositories for all known modules and display them as well.

That invokes the idea of offering RawZips of those modules, since those
are also currently generated on the fly by those pages as well, which I
think might be a political issue. But at least the mention of the
module would be nice to see. Thanks for the great idea. Wanna
implement it? :)

Well, I supposed it's also a political issue for us. We might not want
to list all modules of all known repositories on our page. :)
Jonathan Morgan
2010-11-07 13:43:05 UTC
Permalink
Post by Troy A. Griffitts
Post by Jonathan Morgan
2. Offering a list of downloads at CrossWire tends to suggest that they
are the *only* books available. While I can download zip files from
Xiphos FTP directly (for example), it's not easy to find out about, and
I'm sure some people evaluate the software purely on the basis of the
books available without even trying the software.
Since those pages are already generated dynamically, I suppose we could
simply use our own InstallMgr class to browse all known module
repositories for all known modules and display them as well.
That was indeed my thought.

That invokes the idea of offering RawZips of those modules, since those
Post by Troy A. Griffitts
are also currently generated on the fly by those pages as well, which I
think might be a political issue. But at least the mention of the
module would be nice to see. Thanks for the great idea. Wanna
implement it? :)
I might think about it at some point, but for the moment I really can't take
on any new responsibilities until BPBible 0.5 is completed (I hope I haven't
said that too many times, but if I have just be aware that I'm probably much
more sick of saying it than you are of hearing it). However, I will list a
few "problems" below.

Offering links to Raw Zips (if the particular repository offered them) would
be nice though not necessarily essential.

Well, I supposed it's also a political issue for us. We might not want
Post by Troy A. Griffitts
to list all modules of all known repositories on our page. :)
I agree with this to some extent, but I'm not entirely sure how much
difference there is between showing it on your webpage and showing it in the
InstallMgr of an application (sure, more people will see it, but not
necessarily that many more). Surely you are already using some selection to
determine which repositories can be included?

Anyway, now more on the multiple repository problem.

I understand the vision of multiple repositories and its promise (I think).
However, I think there are a number of problems that can be caused by having
multiple repositories:
1. Automatic updating of installed modules becomes harder: There are more
repositories to download mods.d.tar.gz (or equivalent) from. This
potentially means taking longer, using more resources, etc., and there must
be a point at which you decide this really isn't something we should be
doing.

2. Showing users a view of all books available becomes harder: I think all
the Install Manager implementations I have used end up having one tab per
repository (or something similar) and only showing all the books from that
repository. This makes it much harder for the user to determine whether a
resource is present (for example, why would I click on the Xiphos tab when
looking for StrongsRealGreek? Or for Dore Woodcuts?) I would prefer a
combined view of all of the available modules, and that hits the same
problem with update I mentioned earlier: you potentially need to fetch your
list of available modules from lots of places.

3. Following on from this is the duplicates problem DM mentioned earlier.
At a trivial level it comes up with the NETFree module, which bible.org host
in their own repository, but also asked CrossWire to duplicate it to
increase exposure. If we had a good multi-repository solution then maybe it
would be exposed enough and would just be left to bible.org to host. More
fundamental is the case where the two repositories for whatever reason have
different versions of the same module. DM suggested using KJV (CrossWire)
and KJV (Other Publisher). I personally think there should only be able to
be one installed at a time, in the same way as currently one module with the
same name would be able to be installed in a directory. The reason I think
this is because I get the feeling users don't particularly care which
publisher a book came from: they care what the book is. They ask "What
books are available for your software?", and I think would like one
authoritative list, not "Here is a KJV from publisher X, and here is another
one from publisher Y".

I wonder whether a good solution to the "What books are available in the n
known repositories?" question would be to change the central InstallMgr list
of repositories to not just include a list of blessed repositories, but to
provide a combined mods.d.tar.gz equivalent, which is automatically updated
periodically from all the repositories and ensures applications only have to
check it rather than a potentially unlimited number of repositories. Any
thoughts on this approach? [Ben notes that size of mods.d.tar.gz itself
could end up a problem for responsiveness on slower network connections with
this approach - the CrossWire one is currently 95 KB, and could increase].

These problems are probably almost non-issues with the current number of
repositories. However, if the goal really is lots of publishers hosting
their own repositories I think they are something we might have to think
about. As indicated above, if this is decided to be a good idea I might be
able to help implement it at some point - but probably not this year.

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/6366fb84/attachment.html>
Karl Kleinpaste
2010-11-07 19:37:52 UTC
Permalink
On the one hand, I appreciate a desire for a combined all-repos view of
what's available. It would be useful in some ways. On the other hand,
I find it problematic for several reasons.

- It depends on finding all repos operating. Like it or not, repos go
away for short or long periods, and the first time we hang the user's
module manager because one repo has gone dead for an extended period, we
will cause huge PR problems. Especially as we gain more small
publishers' references in the master repo list, to me the likelihood
that one of these will disappear (if for no other reason than that small
entities will not be good at administering such access) seems high.
Consider the trouble we had some months ago when crosswire.org itself
was firewalled by its ISP to access from all of South Africa and Brazil.
How do we educate users, that they should either manually remove a dead
repo, or must re-sync against the master list when we remove the dead
entry from that? In our one-repo-at-a-time format, as module managers
operate today, at least it's clear which single repo is gone at the
moment, and we don't/can't give an erroneous view that the entire space
of module retrieval has gone dead.

- People know that different stores carry different merchandise. Why is
it any stretch that different repositories have different modules? I
don't expect to find the widest selection of dead-tree Bibles in a
Barnes&Noble, but rather in my local Christian bookstore. Home
construction manuals are best found at Home Depot, not the local
megamall's Borders. (Sure, you'll find some at Borders. The selection
won't be great. B&N and Borders provide "wide but not deep.")

- Jon asks why one would go looking at the Xiphos repo when searching
for modules. It's a good question, but it's just the question of why
one looks in more than one bookstore. The Xiphos repo exists because
(as instantiated some years ago on my home system) I had modules I
wanted to distribute and was not able to get them into the CrossWire
repo. The Xiphos repo distributes approximately 130G/month these days.
Other entities' distribution of modules will be very similar: They have
content that is not appropriate for CrossWire itself. It makes sense to
me that, to find certain publishers' content, one has to go to that
publisher.

This overall multi-store scenario has been perverted (and I mean that in
a proper technical sense) by Amazon. Now everyone expects to find every
book in the world at one place. But at least Amazon is, literally, one
place on the web. Our potentially-many repos are all over, the master
repo list is just our local MapQuest equivalent to find them all, and
each will have widely-varying network support and administrative support.
Jonathan Morgan
2010-11-08 11:35:14 UTC
Permalink
Post by Karl Kleinpaste
On the one hand, I appreciate a desire for a combined all-repos view of
what's available. It would be useful in some ways. On the other hand,
I find it problematic for several reasons.
- It depends on finding all repos operating. Like it or not, repos go
away for short or long periods, and the first time we hang the user's
module manager because one repo has gone dead for an extended period, we
will cause huge PR problems. Especially as we gain more small
publishers' references in the master repo list, to me the likelihood
that one of these will disappear (if for no other reason than that small
entities will not be good at administering such access) seems high.
Consider the trouble we had some months ago when crosswire.org itself
was firewalled by its ISP to access from all of South Africa and Brazil.
How do we educate users, that they should either manually remove a dead
repo, or must re-sync against the master list when we remove the dead
entry from that? In our one-repo-at-a-time format, as module managers
operate today, at least it's clear which single repo is gone at the
moment, and we don't/can't give an erroneous view that the entire space
of module retrieval has gone dead.
Good point: Didn't even occur to me. And I was only reading "The Seven
Fallacies of Distributed Computing" last week :(.

I don't think it's an insuperable obstacle, though. It would be necessary
to make sure that all these error cases have a solution which does not hang
the program, but I don't think that means not doing a combined view.
Post by Karl Kleinpaste
- People know that different stores carry different merchandise. Why is
it any stretch that different repositories have different modules? I
don't expect to find the widest selection of dead-tree Bibles in a
Barnes&Noble, but rather in my local Christian bookstore. Home
construction manuals are best found at Home Depot, not the local
megamall's Borders. (Sure, you'll find some at Borders. The selection
won't be great. B&N and Borders provide "wide but not deep.")
- Jon asks why one would go looking at the Xiphos repo when searching
for modules. It's a good question, but it's just the question of why
one looks in more than one bookstore. The Xiphos repo exists because
(as instantiated some years ago on my home system) I had modules I
wanted to distribute and was not able to get them into the CrossWire
repo. The Xiphos repo distributes approximately 130G/month these days.
Other entities' distribution of modules will be very similar: They have
content that is not appropriate for CrossWire itself. It makes sense to
me that, to find certain publishers' content, one has to go to that
publisher.
Bookstores have physical limitations in what they can carry. We do not
necessarily need to have these limitations.

I still suggest that most people's goals are either:
a. Discover what books are available.
b. Discover if book X is available (or maybe books by author Y or on subject
Z?)

If faced with a current Install Manager, I would definitely open
repositories one after another to try and answer these questions, but if
there is a better way to meet that goal (such as showing a combined view)
why not?

I tend to dislike software that forces me to search in certain ways: whether
it's "You must select the language before we show you what's available" or
"You must select the type of book" or "You must select the publisher's
repository", there will be some times when this matches the way I want to
search and what I'm looking for, and some times when it does not.

This overall multi-store scenario has been perverted (and I mean that in
Post by Karl Kleinpaste
a proper technical sense) by Amazon. Now everyone expects to find every
book in the world at one place. But at least Amazon is, literally, one
place on the web. Our potentially-many repos are all over, the master
repo list is just our local MapQuest equivalent to find them all, and
each will have widely-varying network support and administrative support.
Interesting thoughts. However, my expectation comes not from Amazon, but
from other Bible software I have used. If I want to know which books are
available for e-Sword, I go to the e-Sword website. If I want to know which
books are available for Logos, I go to the Logos website. If I want to know
which books are available for CrossWire, I go to the CrossWire website. It
doesn't list Dore Woodcuts, so I know it doesn't support it. [I do actually
know that both e-Sword and Logos have resources which aren't sold by the
respective company, but with Logos it's a minority and with e-Sword my
impression is that they are mostly breaking copyright so I don't bother
looking]. Because I'm interested in Bible software, I'm usually willing to
spend the time looking. I'm not sure that everyone is.

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/c0667c72/attachment.html>
Karl Kleinpaste
2010-11-08 12:57:51 UTC
Permalink
whether it's "You must select the language before we show you what's
available" or "You must select the type of book" or "You must select
the publisher's repository", there will be some times when this
matches the way I want to search and what I'm looking for, and some
times when it does not.
I find it an extraordinarily odd perspective, that knowing what type of
book you're looking for is not a normal, all-the-time prerequisite to
finding what you want. Does anyone ever ask a salesclerk, whether human
or automated, "show me the book selection, but I don't even know what
I'm looking for, much less the language in which it's printed, least of
all the author or publisher"?

Since this is Bible study software, how would it ever be, that someone
looking for new books in our environment would not know if he's looking
for a Bible, or a commentary, or something else?

If my imagination is insufficient about this problem, feel free to say
so. But the idea that you have to specify that you need English or
Suomi or Caribbean Javanese texts seems to me to be a "level 0" kind of
requirement.

In any event, as a current use case, Xiphos' module manager provides
module listings both per-category as well as per-availability (that is,
updates and yet-uninstalled content). Under each per-availability case,
there is only a language distinction, on the idea that you still want to
hunt down modules in a language you can read, but there is no type
(Bible, commentary, genbook, ...) distinction. Users have found it
productive to have the ability to locate everything available in
English, for example.
Interesting thoughts. However, my expectation comes not from Amazon,
but from other Bible software I have used. If I want to know which
books are available for e-Sword, I go to the e-Sword website. If I
want to know which books are available for Logos, I go to the Logos
website. If I want to know which books are available for CrossWire, I
go to the CrossWire website. It doesn't list Dore Woodcuts, so I know
it doesn't support it. [I do actually know that both e-Sword and
Logos have resources which aren't sold by the respective company, but
with Logos it's a minority and with e-Sword my impression is that they
are mostly breaking copyright so I don't bother looking]. Because I'm
interested in Bible software, I'm usually willing to spend the time
looking. I'm not sure that everyone is.
I think the problem is that the model breaks down in the face of our
intent for multiple publishers. Amazon gives us the
one-place-in-the-world model. You describe Logos and eSword in the same
motif. But that's expressly not what we're up against. We're trying to
create an MxN crossbar, users x publishers. We're not (and never will
be) in bed with publishers in the same way that Logos and eSword are.

We may need another model entirely.
Jonathan Morgan
2010-11-08 13:09:29 UTC
Permalink
Post by Karl Kleinpaste
whether it's "You must select the language before we show you what's
available" or "You must select the type of book" or "You must select
the publisher's repository", there will be some times when this
matches the way I want to search and what I'm looking for, and some
times when it does not.
I find it an extraordinarily odd perspective, that knowing what type of
book you're looking for is not a normal, all-the-time prerequisite to
finding what you want. ?Does anyone ever ask a salesclerk, whether human
or automated, "show me the book selection, but I don't even know what
I'm looking for, much less the language in which it's printed, least of
all the author or publisher"?
Since this is Bible study software, how would it ever be, that someone
looking for new books in our environment would not know if he's looking
for a Bible, or a commentary, or something else?
If my imagination is insufficient about this problem, feel free to say
so. ?But the idea that you have to specify that you need English or
Suomi or Caribbean Javanese texts seems to me to be a "level 0" kind of
requirement.
If I'm looking for the ESV, it's entirely irrelevant which language
it's in (yes, I know it's English, but rather than think it's English
I could just type ESV). Similarly, if I'm looking for a particular
author I want all books by that author. They have probably written
only in one language, so the language choice is irrelevant. If I'm
looking for a particular subject, I may be able to filter down to that
subject and then scan through the results irrespective of whether they
are in multiple languages or book types.
Post by Karl Kleinpaste
In any event, as a current use case, Xiphos' module manager provides
module listings both per-category as well as per-availability (that is,
updates and yet-uninstalled content). ?Under each per-availability case,
there is only a language distinction, on the idea that you still want to
hunt down modules in a language you can read, but there is no type
(Bible, commentary, genbook, ...) distinction. ?Users have found it
productive to have the ability to locate everything available in
English, for example.
Indeed.
Post by Karl Kleinpaste
Interesting thoughts. ?However, my expectation comes not from Amazon,
but from other Bible software I have used. ?If I want to know which
books are available for e-Sword, I go to the e-Sword website. ?If I
want to know which books are available for Logos, I go to the Logos
website. ?If I want to know which books are available for CrossWire, I
go to the CrossWire website. ?It doesn't list Dore Woodcuts, so I know
it doesn't support it. ?[I do actually know that both e-Sword and
Logos have resources which aren't sold by the respective company, but
with Logos it's a minority and with e-Sword my impression is that they
are mostly breaking copyright so I don't bother looking]. ?Because I'm
interested in Bible software, I'm usually willing to spend the time
looking. ?I'm not sure that everyone is.
I think the problem is that the model breaks down in the face of our
intent for multiple publishers. ?Amazon gives us the
one-place-in-the-world model. ?You describe Logos and eSword in the same
motif. ?But that's expressly not what we're up against. ?We're trying to
create an MxN crossbar, users x publishers. ?We're not (and never will
be) in bed with publishers in the same way that Logos and eSword are.
We may need another model entirely.
Agreed, which is why I suggest such a different model which aims to
still have as many of the advantages of the one publisher model as it
can.

Jon
Peter von Kaehne
2010-11-08 14:50:00 UTC
Permalink
On Mon, 08 Nov 2010 07:57:51 -0500
Post by Karl Kleinpaste
whether it's "You must select the language before we show you what's
available" or "You must select the type of book" or "You must select
the publisher's repository", there will be some times when this
matches the way I want to search and what I'm looking for, and some
times when it does not.
I find it an extraordinarily odd perspective, that knowing what type
of book you're looking for is not a normal, all-the-time prerequisite
to finding what you want. Does anyone ever ask a salesclerk, whether
human or automated, "show me the book selection, but I don't even
know what I'm looking for, much less the language in which it's
printed, least of all the author or publisher"?
Karl, not by necessity.

1) Some of our Genbooks could easily be formatted as a commentary.
Probably it also should be done and that would remove one reason to look
somewhere else. But right now the distinction is not totally a sharp
one. Maybe it would be worthwhile to file a bug against modules which
offend in this fashion. Luther's complete works is one like that as it
the Father's module (both in parts).

2) Language - I use a lot of Martin Luther's and Calvin's stuff at the
moment for bible college work. I honestly do not care which language it
is as long as it is one I can read. And not all that should be
available in German is available. In fact our English language Luther
offerings are way better than our German ones. So I use them.
Post by Karl Kleinpaste
Since this is Bible study software, how would it ever be, that someone
looking for new books in our environment would not know if he's
looking for a Bible, or a commentary, or something else?
If my imagination is insufficient about this problem, feel free to say
so. But the idea that you have to specify that you need English or
Suomi or Caribbean Javanese texts seems to me to be a "level 0" kind
of requirement.
As above. Mostly yes, but not always. Author could be occasionally more
relevant.

In fact, I find the language distinction occasionally quite
annoying. I have 4 bibles I use and they are the ESV, the Schlachter
2000, a Farsi one and Luther. I read them for what they are - different
translations with different use cases for me or different emphasis.

I also do have a multitude of other installs, but they are used only
occasionally and yes, there, language choice is probably relevant, but
- I am a module developer. I am bound to have anything and everything
available even stuff I can not read, let alone understand. Most users
are probably not.

I think from a usability POV, I am wondering if the language choice
could not be something which should be possible to switch on and off.
Our hypothetical user might be

- a minister - own language and original languages + maybe French or
English.
- a monolingual Anglo-Saxon/German/whatnot user - hide all that is not
for him.
- a missionary needing his own language, maybe English or French and a
couple of other languages +/- original languages
- a Bible Software enthusiast who gets mental raptures for seeing a
Bible in Southern Timbuctulese available (never mind that he knows
not a single soul who could read that), he needs to have access to
all, but is in fact not our main target user.

But - apart from the last user for whom I think we do not real should
consider - most users will eventually have a selection of 3-10 Bibles
which mentally will not get selected for reading for their language but
simply for what they are - i.e. by name.

I know that I have in the past asked for language tree selection and I
stand by that - it is way better than what we had prior where you had to
dig about to find what you wanted - and then realised that most stuff
was in English.

But now I think the Central-American BTI bibles do a bit my head in
every time I have to wade past them...

This is no objection to anything else said by anyone.

Peter
Brian J. Dumont
2010-11-08 15:12:05 UTC
Permalink
Post by Peter von Kaehne
1) Some of our Genbooks could easily be formatted as a commentary.
Probably it also should be done and that would remove one reason to look
somewhere else. But right now the distinction is not totally a sharp
one. Maybe it would be worthwhile to file a bug against modules which
offend in this fashion. Luther's complete works is one like that as it
the Father's module (both in parts).
FYI, there *are* two modules that have significant Luther content:
LuthersWorks (genbook of as much PD content as can be gathered) and
Luther (commentary form of specifically commentary material). I am
still actively updating both.

Pie-in-the-sky plans are to do the same for the EarlyFathers module, but
that's at least a year off I'd say.

Enjoy,
Brian
DM Smith
2010-11-09 04:15:34 UTC
Permalink
I tend to dislike software that forces me to search in certain ways: whether it's "You must select the language before we show you what's available" or "You must select the type of book" or "You must select the publisher's repository", there will be some times when this matches the way I want to search and what I'm looking for, and some times when it does not.
You might like how MacSword does it. It presents all the works from a repository as a grid with a navigation tree. Clicking on any level in the tree prunes the grid. Clicking on a heading sorts that column. There is a search mechanism that searches the "initials".

There is a lot to like about it.

In Him,
DM
Peter von Kaehne
2010-11-05 13:02:58 UTC
Permalink
I think Troy, the concern is correct.

For the publisher with some decent IT muscle and budget a proper repo must be better, but for the small town church with a website and a couple of modules to share - zip and http is a must.

Having a multiplicity of methods of getting modules into the system would certainly be easier.

My preference:

1) keep current methods - it is best for huge numbers of modules and it is probably also best for anyone with enough money to have a fixed ip and a server, able to run anonymous ftp

2) add methods for local installation of zips. Look at MK Bible - pull a zip over the programme and it gets installed. This is - emphatically - not how I would want to install a large selection of modules or how I would want to publish the same, but it is probably the best usability i have seen for a frontend using only 2-3 modules (which is what MKBible is laid out for) and a small time publisher or someone who has target audience of little computer literacy

Peter
Von: admin at bible.salterrae.net
An: "SWORD Developers\' Collaboration Forum" <sword-devel at crosswire.org>
Betreff: Re: [sword-devel] Remote Module Repository Wiki
Post by Troy A. Griffitts
I do understand the very real practical concerns about FTP you point
out, but we have practically received almost no support emails from
users not being able to install because FTP was blocked. Most hosting
services provide FTP access. The issue we ran into before was some
didn't provide anonymous ftp access, which we've rectified by adding
username / password fields in our repository registry and support for
this in our installer, and in reality to install an FTP server on a
windows box is trivial compared to the ongoing maintenance of assuring
zips are created and placed in the correct folder, an index file for the
zips is created and kept updated whenever anything changes, etc.
I don't think that installing an FTP server is trivial.
I can't agree that 'most hosting services provide FTP access.'
Most cheep hosting services provide ftp access only for its maintainer,
ftp access is not permitted for public use.
HTTP services are name-based , FTP services are IP-based. So
if one want to service an FTP server, one must own one IP address
for it, contrary to the fact that many HTTP services can share one
IP address.
Furthermore FTP service needs two connections , one for control and
one for data. one is outgoing, another is incoming. this make
firewall rules very complexing.
Servicing an HTTP server is easy, and servicing an FTP server is annoying.
If both methods can serve same jobs, I would rather choose HTTP .
Post by Troy A. Griffitts
It might seem trivial to us, and might actually be trivial for the
publishers, in practice (and might eventually be how we end up going if
--
admin at bible.salterrae.net
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
GMX DSL Doppel-Flat ab 19,99 &euro;/mtl.! Jetzt auch mit
gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl
Peter von Kaehne
2010-11-05 15:09:33 UTC
Permalink
Post by Troy A. Griffitts
Is there any chance I can get some of you who have contributed to this
valuable resource to clean it up a bit.
its size reduced as small as possible to show how easy it is to run a
remote repository. Initial, "simple repository" section greatly
appreciated, but the rest is still a little cloudy for even me to
understand.
To come back to the original request

1) I have removed all references to MacOS zip and Windows zips from the
wiki.

2) I have removed all references to Jsword needing special treatment
from the wiki

3) I have mildly simplified the explanations.

Please check it for correctness and do any further improvements as
requested by Troy

Peter
David Haslam
2010-11-05 08:57:49 UTC
Permalink
Please respond to the original message.

Troy's reply on a separate topic has somehow got linked to this thread, and
I don't wish to have my report overlooked.

David
--
View this message in context: http://sword-dev.350566.n4.nabble.com/Alternative-method-to-mark-translation-changes-tp3027931p3028315.html
Sent from the SWORD Dev mailing list archive at Nabble.com.
Troy A. Griffitts
2010-11-05 10:13:50 UTC
Permalink
Unless I've forgotten about this-- which is entirely possible-- I had no
idea we had transChange marked up in userspace tags in any of our
modules. I consider this a bug of either:

our modules for not using proper OSIS <transChange> tags, or
OSIS for not providing a way to legitimately markup <w> and
<transChange> together

But, until I have time to thoroughly review the individual cases which
apparently require userspace tags to mark these transChange occurrences,
I cannot make a judgment. I do know that DM is a smart guy and
generally give him the benefit of the doubt :) So I'm inclined to blame
OSIS, but have no factual evidence for my accusation.

Troy
Post by David Haslam
Please respond to the original message.
Troy's reply on a separate topic has somehow got linked to this thread, and
I don't wish to have my report overlooked.
David
David Haslam
2010-11-05 09:22:44 UTC
Permalink
Karl Kleinpaste has kindly responded via Xiphos tracker as follows,

xiphos has nothing to say about this aspect of display. xiphos simply
requests content from the engine, and whatever comes back is what goes into
the display. if it displays improperly, it is a filter bug in the engine by
failing to include html tags ...Ergo: I think I've discovered a bug in the
SWORD engine.

David
--
View this message in context: http://sword-dev.350566.n4.nabble.com/Alternative-method-to-mark-translation-changes-tp3027931p3028351.html
Sent from the SWORD Dev mailing list archive at Nabble.com.
Continue reading on narkive:
Loading...