[Chapter 3] 3.4 Aggregating Web content in newsgroups

3.4 Aggregating Web content in newsgroups

When the Microsoft and Netscape messaging clients display plain ASCII messages, they automatically activate URLs found in those messages. If I compose such a message, using a messaging client that's set to text mode (see tip below), I need only mention the site http://udell.roninhouse.com/ in a message that I post to a newsgroup, or email to you. Your message reader will render the URL's text as a clickable hyperlink. By merely reproducing a correctly-spelled URL, we become - in a limited but important sense - hypertext authors. In the text-mode messaging environment, nobody has to know that the HTML representation of that link is <a href="http://udell.roninhouse.com/">http://udell.roninhouse.com</a>. You can just type an URL, or better yet cut and paste one into your message.

Usage Tip: Text mode versus HTML mode in message composers

The Microsoft and Netscape messaging clients can compose either in text mode, or HTML mode. People are most familiar with text mode. It produces messages whose bodies are just lines of ASCII text, typically not longer than 65 or 70 characters. Alternatively these clients can operate in HTML mode. In Outlook Express, you turn on HTML mode using Tools->Options->Send->Mail Sending Format (or ->News Sending Format); the choices are HTML and Plain Text. In Collabra, you use Edit->Preferences->Mail & Newsgroups->Formatting; the choices are "Use the HTML editor" and "Use the plain text editor."

It's true that results aren't always perfect. Long URLs, especially CGI-style URLs containing characters such as the question mark and the ampersand, often run afoul of the message reader's line-wrapping algorithm and fail to render as clickable links (see tip below). Still, this method of URL autoactivation has made a huge impact on the world. Text mode is the lowest common denominator of Internet messaging. Millions of authors of simple ASCII email messages and newsgroup postings now routinely use live citations to other texts. To a literate person from an earlier era, or even to all of us just a decade ago, this would have seemed miraculous. Indeed it is. As we communicate in the context of the Web, we can create new views of it.

UsageTip: Bracketing URLs in text-mode messages

RFC 1738 suggests that <URL:http://udell.roninhouse.com/> should signal to a messaging client that the indicated URL should be rendered as a single unbroken entity. In fact URLs written this way don't always work reliably in messaging clients. Outlook Express, for example, won't autoactivate them. Many people instead use the style <http://udell.roninhouse.com/> to delimit an URL. This style of bracketed URL does tend to survive the line-oriented ASCII environment much better than the unbracketed style.

Neither approach works reliably in all messaging clients all the time. What to do? Remember, this is only a problem in text mode. In HTML mode, you can produce shorter, more descriptive URLs that will always work reliably when viewed by HTML-aware messaging clients.

A great many Web pages do nothing but aggregate other Web pages. Thanks to URL autoactivation, even plain ASCII email messages and newsgroup postings can do the same. When you dash off a message that includes a handful of URL citations, you're constructing a unique view of the Web to support your message. In many Usenet newsgroups this happens on a grander scale. A FAQ maintainer will periodically repost a message that documents some area of knowledge and, in so doing, aggregate relevant Web content. These documents behave like published bookmark files.

3.4.1 Newsgroups as shared, annotated bookmark files

In local newsgroups you can take this notion of a published bookmark file a few steps further. Remember that message Bob posted, citing an URL for some LDAP-related documentation? Let's suppose that Bob's team owns a newsgroup called webteam.bookmarks. In this newsgroup, every top-level posting works like a bookmark folder, and contains a set of online resources for subjects like LDAP, SSL, PGP, and so on. Like Usenet FAQs but unlike browser bookmarks, these postings can annotate the URLs they record. A project team that consolidates its online research in this way can do more than centralize the gathering of bookmarks. It can also pool its knowledge about the nature and value of those bookmarks. Team members can, for example, build a consensus as to which sources are most valuable, and why. Critical evaluation of the quality of online resources is the price we all pay for easy access to so many of them. That analysis takes an investment of effort; groupware should help us maximize our return on that investment.

Because Usenet FAQs expire, their authors periodically repost them. This recycling mechanism creates natural opportunities to update these documents. But how should updates be handled in a local newsgroup where messages live forever? One approach is to cancel the message, and post an update. Because newsreaders only let you cancel your own messages, this method presumes that the original author of the message is also its maintainer. Another approach is to post updates as responses to the original message. That way, anyone can contribute an update. The set of responses to a message creates an audit trail that documents who posted updates and when.

I prefer the second approach because anyone can add new material or comment on what's already there. Admittedly it's not perfect. In a hypothetical WebDAV-based conferencing system, for example, regions of a document will be locked and edited in-situ. And an audit trail will be captured more subtly than as a set of response messages. What will we do until that technically superior solution arrives? If a team is committed to pooling the Web resources it collects - a very big if, to be sure - the technique I've described here can be a simpler and easier alternative to intranet Web pages that serve the same purpose.

3.4.2 Including Web content in newsgroups

The technique of aggregation by reference - that is, assembling URLs that refer to Web pages - isn't flawless. A remote server may not be available when you need it. If it is, the page you want may no longer be there. More importantly, although your annotations will be searchable if your newsgroup is searchable, the referenced content won't integrate with your local search engine. It can therefore be useful to add remote documents to your local site's collection. In Collabra's message composer, called Composer, the File->Attach->Web Page option enables you to enter an URL whose content will be attached to your message. You can type the URL, but it's better to surf to the page to ensure that it exists, then copy the URL from the browser's Location: field to the clipboard and paste it into Composer's attachment dialog box. Easiest of all, if you're browsing a Web page that contains your target URL in hyperlink form, you can drag that link into Composer's attachment pane. This little-known procedure is extremely handy!

For Outlook Express users, the procedure is more cumbersome. You have to save a Web page as a file, then attach the file using Edit->Insert->File Attachment. If you drag a link from the browser and drop it into the message composer, Outlook Express appears to attach the page. But that attachment is actually just the URL itself, not the page pointed to by the URL.

Web pages aren't the only kinds of documents you might want to move into your local collection. Sometimes it makes sense to do the same thing with downloadable files. For example, it's always a challenge to keep up with the current versions of system-software components, applications, drivers, and related documentation. Vendors often make these available for download from their sites, but an aggregated and annotated view of the items that matter to your group can save time and effort. In Collabra, the File->Attach->Web Page technique will, in a single action, download a remote binary file and add it as an attachment to your posting. In Outlook Express, it's a two-stroke operation. First you download the file to your local disk, then you attach it. In this scenario, the newsgroup provides more than a richly-annotated alternative to the LAN file system. It combines a means to aggregate these files with a means to discuss them. If you're going to install a driver, you'd like to know what someone else learned while doing so. Newsgroups encourage informal discussion that can weave in and around an annotated data store.

In this context, local means file-system-accessible rather than Web-accessible, and the scope of the file system includes both your local drive and also all visible network drives. A file in any of these places can be uploaded as an attachment to a newsgroup posting. Netscape users: File->Attach->File. Microsoft users: Insert->File Attachment.

Sometimes it makes sense to transform a document that you include in a posting. Consider a Microsoft Word file that documents a new driver release. If you attach it (or refer to it) the reader of your posting will be presented with a link that, when clicked, invites that person to save the file for later use in Word, or else open it directly in Word. Anyone who doesn't have Word (and the right version of Word!) is out of luck. Even when Word is installed, it may not be running. In this case you force the reader of your posting to wait for Word to launch, then load the attached file.

Alternatively you could use Word's Save As HTML feature to convert the .DOC file to a .HTML file, and then attach that. The obvious benefit is that the content will be instantly available to everyone, requiring no viewer except the newsreader itself. (Nothing prevents you from also attaching the .DOC, for those users who prefer Word, or to ensure faithful transmission of features of the document that may not survive conversion to HTML.) There is also a subtler benefit to this approach. When you attach a binary file to a newsgroup (or, for that matter, to an email message) it will be encoded (technically: base64-encoded or MIME-encoded) as a stream of apparently meaningless ASCII text. Text in the original document won't be recognizable in the encoded version. This makes fulltext indexing and search futile, even when indexers can work with non-ASCII formats such as Word's .DOC or Acrobat's .PDF. So, in search-enabled newsgroups, it's a good idea to convert binary word-processor file formats to HTML, in order to expose the content to the indexer.

3.4.3 Referring to newsgroup messages

When news servers don't expire messages, they persist indefinitely. Each posting produces a news URL - something like news://udell.roninhouse.com/358C707B.ED39B9C1@monad.net - that's unique and that can be used to retrieve the message months or even years later. Why do news hyperlinks matter? For starters, they're a better way to quote prior messages. Wholesale quoting of a message into a reply to that message is a much-abused practice that carries over from the Usenet to private conferencing. It's often useful to intersperse commentary with quoted bits of a prior message, when that commentary applies specifically to the cited material. When a response merely refers to a prior message in its entirety, though, there is no need to quote the whole antecedent. Let the response hierarchy show how your message relates to its antecedent. Or if that relationship is distant, cite the news:// URL of the antecedent. These kinds of hyperlinks can add a new dimension to corporate knowledge management. Users can not only contribute documents to a shared archive, but also weave interconnections among those documents.

How do you cite one news message when composing another? Collabra solves this problem handily. You can drag the icon representing the message you want to cite into the message composer's window. If the composer is set to text mode, the raw link address (e.g. news://udell.roninhouse.com/358C707B.ED39B9C1@monad.net) will appear. URL autoactivation will make it active when viewed in a newsreader. If instead the composer is set to HTML mode, a link label (e.g. LDIF specification) will appear. This label, which echoes the Subject: header of the cited message, becomes the visible part of a hyperlink which encodes the raw link address as its HREF attribute. The newly-composed HTML message will, when viewed in an HTML-aware newsreader, display the label as a clickable link, just as it might appear on a Web page.

Collabra's drag-and-drop hypertext authoring capability is remarkable, though scarcely known. Just about anything represented by an URL - including Web pages and newsgroup messages on any server - can be cited in a news (or email) message with very little effort. In a local newsgroup setting, this greatly enhances the ability of the newsgroup to function like a Web server. At one point, for example, I posted to a staffwide newsgroup a long memo describing how users could manage their own accounts on a mail server I maintained. Weeks later, people needing that information were asking - in the newsgroup, and by email - how to find it. I answered those questions by referring to the original posting. Without a hyperlink, that would have been cumbersome: "Scan bytestaff.operations, sometime in March, look for a message from me titled Email Account Info." But Collabra's hyperlink-authoring feature made it a snap. I dragged that message onto my desktop for instant access, and could then simply drop it (that is, drop its URL) into my responses. The same method would have been available if I'd put the document on our intranet Web server. But it was even easier to post it to the staff newsgroup, and doing that killed two birds with one stone. It announced the memo to the group, and also created a linkable document for future reference.

In Outlook Express, it's much harder to use hyperlinks to refer to news content. You can drag a message icon or a message hyperlink into the composer's window, but doing that doesn't create a hyperlink. Instead it attaches the dropped URL to the newly-composed message. To compose a link, you have to use the Create a Hyperlink button that appears on the Composer's toolbar when it's in HTML mode. It asks for a link type (e.g. http:, news:, ftp:) and a link address. Then you need to plug in the address. Many users will understand how to do that for a Web page: visit the page, copy its URL, and paste it in as the link address. But few will ever discover how to do the same for a news message. Why? A news:// URL is formed from an NNTP message ID. Outlook Express deems these IDs unfriendly, and hides them from the user. It is possible to expose a message's ID, and copy it as the address of a newly-composed hyperlink. But the procedure is arcane and cumbersome. Virtually no-one will discover it, and even those who do won't be able to use it quickly or easily. If you're curious, here's the drill: right-click a message icon to reveal its context menu. Do Properties->Details. Select the angle-bracketed text following the Message-ID: header. Use the keyboard copying method (e.g. Control-C in Windows) to transfer the message ID to the clipboard. Then select the link address input box, and use the keyboard paste method (e.g. Control-V in Windows) to paste the message ID as the address of the link.

3.4.4 Following newsgroup hyperlinks

Collabra's more aggressive approach to hypertext authoring means that Netscape users are best-equipped to crosslink content archived to local newsgroups. Once a news:// hyperlink is created, though, both Collabra and Outlook Express can follow it. In Collabra, clicking such a link replaces the source message that contains the link with the target message specified by the link's address. To restore the source message, use Go->Back. All this occurs in the context of a single message-reading window. In Outlook Express, clicking a message link launches a new message-reading window and displays the target message in it.

You can also include news:// URLs in Web pages. Why do that? As we saw in the last chapter, a newsgroup can function as the raw material for a more formal document, such as a FAQ. Using news hyperlinks, a FAQ published on your intranet can both summarize and connect to a supporting newsgroup.

The behavior of a news:// URL on a Web page varies by client. When you click on such an URL in either Navigator or Internet Explorer, the mail/news message-reader will launch and display the message. In Communicator this transition from Web space to news space is flawed. You can read the message, but not reply to it. Outlook Express, as of MSIE 4.01, gets this right - you can reply to a message that you jump to by way of a news:// URL. However both products fail to achieve the next level of Web/news integration. Ideally a news:// URL accessed from a Web page should not only display the individual message, but also establish a newsreader context surrounding that message - for example, by launching the newsreader, locating the newsgroup, and highlighting the message in the message-list pane. Lacking this feature, both products fail to maximize the potential synergy between the Web and news domains.


	Practical Internet Groupware By Jon Udell 1st Edition October 1999 (est.) 1-56592-537-8, Order Number: 5378 384 pages (est.), $29.95 (est.)


3.3 The quest for a read/write Web server		3.5 HTML authoring strategies

Practical Internet Groupware

3.4 Aggregating Web content in newsgroups

3.4.1 Newsgroups as shared, annotated bookmark files

3.4.2 Including Web content in newsgroups

3.4.3 Referring to newsgroup messages

3.4.4 Following newsgroup hyperlinks