1 2 3 4 5 6 7Network Working Group M. Rose 8Request for Comments: 1082 TWG 9 November 1988 10 11 12 13 Post Office Protocol - Version 3 14 Extended Service Offerings 15 16Status of This Memo 17 18 This memo suggests a simple method for workstations to dynamically 19 access mail from a discussion group server, as an extension to an 20 earlier memo which dealt with dynamically accessing mail from a 21 mailbox server using the Post Office Protocol - Version 3 (POP3). 22 This RFC specifies a proposed protocol for the Internet community, 23 and requests discussion and suggestions for improvements. All of the 24 extensions described in this memo to the POP3 are OPTIONAL. 25 Distribution of this memo is unlimited. 26 27Introduction and Motivation 28 29 It is assumed that the reader is familiar with RFC 1081 that 30 discusses the Post Office Protocol - Version 3 (POP3) [RFC1081]. 31 This memo describes extensions to the POP3 which enhance the service 32 it offers to clients. This additional service permits a client host 33 to access discussion group mail, which is often kept in a separate 34 spool area, using the general POP3 facilities. 35 36 The next section describes the evolution of discussion groups and the 37 technologies currently used to implement them. To summarize: 38 39 o An exploder is used to map from a single address to 40 a list of addresses which subscribe to the list, and redirects 41 any subsequent error reports associated with the delivery of 42 each message. This has two primary advantages: 43 - Subscribers need know only a single address 44 - Responsible parties get the error reports and not 45 the subscribers 46 47 48 49 50 51 52 53 54 55 56 57 58Rose [Page 1] 59 60RFC 1082 POP3 Extended Service November 1988 61 62 63 o Typically, each subscription address is not a person's private 64 maildrop, but a system-wide maildrop, which can be accessed 65 by more than one user. This has several advantages: 66 - Only a single copy of each message need traverse the 67 net for a given site (which may contain several local 68 hosts). This conserves bandwidth and cycles. 69 - Only a single copy of each message need reside on each 70 subscribing host. This conserves disk space. 71 - The private maildrop for each user is not cluttered 72 with discussion group mail. 73 74 Despite this optimization of resources, further economy can be 75 achieved at sites with more than one host. Typically, sites with 76 more than one host either: 77 78 1. Replicate discussion group mail on each host. This 79 results in literally gigabytes of disk space committed to 80 unnecessarily store redundant information. 81 82 2. Keep discussion group mail on one host and give all users a 83 login on that host (in addition to any other logins they may 84 have). This is usually a gross inconvenience for users who 85 work on other hosts, or a burden to users who are forced to 86 work on that host. 87 88 As discussed in [RFC1081], the problem of giving workstations dynamic 89 access to mail from a mailbox server has been explored in great 90 detail (originally there was [RFC918], this prompted the author to 91 write [RFC1081], independently of this [RFC918] was upgraded to 92 [RFC937]). A natural solution to the problem outlined above is to 93 keep discussion group mail on a mailbox server at each site and 94 permit different hosts at that site to employ the POP3 to access 95 discussion group mail. If implemented properly, this avoids the 96 problems of both strategies outlined above. 97 98 ASIDE: It might be noted that a good distributed filesystem 99 could also solve this problem. Sadly, "good" 100 distributed filesystems, which do not suffer 101 unacceptable response time for interactive use, are 102 few and far between these days! 103 104 Given this motivation, now let's consider discussion groups, both in 105 general and from the point of view of a user agent. Following this, 106 extensions to the POP3 defined in [RFC1081] are presented. Finally, 107 some additional policy details are discussed along with some initial 108 experiences. 109 110 111 112 113 114Rose [Page 2] 115 116RFC 1082 POP3 Extended Service November 1988 117 118 119What's in a Discussion Group 120 121 Since mailers and user agents first crawled out of the primordial 122 ARPAnet, the value of discussion groups have been appreciated, 123 (though their implementation has not always been well-understood). 124 125 Described simply, a discussion group is composed of a number of 126 subscribers with a common interest. These subscribers post mail to a 127 single address, known as a distribution address. From this 128 distribution address, a copy of the message is sent to each 129 subscriber. Each group has a moderator, which is the person that 130 administrates the group. The moderator can usually be reached at a 131 special address, known as a request address. Usually, the 132 responsibilities of the moderator are quite simple, since the mail 133 system handles the distribution to subscribers automatically. In 134 some cases, the interest group, instead of being distributed directly 135 to its subscribers, is put into a digest format by the moderator and 136 then sent to the subscribers. Although this requires more work on 137 the part of the moderator, such groups tend to be better organized. 138 139 Unfortunately, there are a few problems with the scheme outlined 140 above. First, if two users on the same host subscribe to the same 141 interest group, two copies of the message get delivered. This is 142 wasteful of both processor and disk resources. 143 144 Second, some of these groups carry a lot of traffic. Although 145 subscription to an group does indicate interest on the part of a 146 subscriber, it is usually not interesting to get 50 messages or so 147 delivered to the user's private maildrop each day, interspersed with 148 personal mail, that is likely to be of a much more important and 149 timely nature. 150 151 Third, if a subscriber on the distribution list for a group becomes 152 "bad" somehow, the originator of the message and not the moderator of 153 the group is notified. It is not uncommon for a large list to have 154 10 or so bogus addresses present. This results in the originator 155 being flooded with "error messages" from mailers across the Internet 156 stating that a given address on the list was bad. Needless to say, 157 the originator usually could not care less if the bogus addresses got 158 a copy of the message or not. The originator is merely interested in 159 posting a message to the group at large. Furthermore, the moderator 160 of the group does care if there are bogus addresses on the list, but 161 ironically does not receive notification. 162 163 There are various approaches which can be used to solve some or all 164 of these problems. Usually these involve placing an exploder agent 165 at the distribution source of the discussion group, which expands the 166 name of the group into the list of subscription addresses for the 167 168 169 170Rose [Page 3] 171 172RFC 1082 POP3 Extended Service November 1988 173 174 175 group. In the process, the exploder will also change the address 176 that receives error notifications to be the request address or other 177 responsible party. 178 179 A complementary approach, used in order to cut down on resource 180 utilization of all kinds, replaces all the subscribers at a single 181 host (or group of hosts under a single administration) with a single 182 address at that host. This address maps to a file on the host, 183 usually in a spool area, which all users can access. (Advanced 184 implementations can also implement private discussion groups this 185 way, in which a single copy of each message is kept, but is 186 accessible to only a select number of users on the host.) 187 188 The two approaches can be combined to avoid all of the problems 189 described above. 190 191 Finally, a third approach can be taken, which can be used to aid user 192 agents processing mail for the discussion group: In order to speed 193 querying of the maildrop which contains the local host's copy of the 194 discussion group, two other items are usually associated with the 195 discussion group, on a local basis. These are the maxima and the 196 last-date. Each time a message is received for the group on the 197 local host, the maxima is increased by at least one. Furthermore, 198 when a new maxima is generated, the current date is determined. This 199 is called the last date. As the message is entered into the local 200 maildrop, it is given the current maxima and last-date. This permits 201 the user agent to quickly determine if new messages are present in 202 the maildrop. 203 204 NOTE: The maxima may be characterized as a monotonically 205 increasing quanity. Although sucessive values of the 206 maxima need not be consecutive, any maxima assigned 207 is always greater than any previously assigned value. 208 209Definition of Terms 210 211 To formalize these notions somewhat, consider the following 7 212 parameters which describe a given discussion group from the 213 perspective of the user agent (the syntax given is from [RFC822]): 214 215 216 217 218 219 220 221 222 223 224 225 226Rose [Page 4] 227 228RFC 1082 POP3 Extended Service November 1988 229 230 231 NAME Meaning: the name of the discussion group 232 Syntax: TOKEN (ALPHA *[ ALPHA / DIGIT / "-" ]) 233 (case-insensitive recognition) 234 Example: unix-wizards 235 236 ALIASES Meaning: alternates names for the group, which 237 are locally meaningful; these are 238 typically used to shorten user typein 239 Syntax: TOKEN (case-insensitive recognition) 240 Example: uwiz 241 242 ADDRESS Meaning: the primary source of the group 243 Syntax: 822 address 244 Example: Unix-Wizards@BRL.MIL 245 246 REQUEST Meaning: the primary moderator of the group 247 Syntax: 822 address 248 Example: Unix-Wizards-Request@BRL.MIL 249 250 FLAGS Meaning: locally meaningful flags associated 251 with the discussion group; this memo 252 leaves interpretation of this 253 parameter to each POP3 implementation 254 Syntax: octal number 255 Example: 01 256 257 MAXIMA Meaning: the magic cookie associated with the 258 last message locally received for the 259 group; it is the property of the magic 260 cookie that it's value NEVER 261 decreases, and increases by at least 262 one each time a message is locally 263 received 264 Syntax: decimal number 265 Example: 1004 266 267 LASTDATE Meaning: the date that the last message was 268 locally received 269 Syntax: 822 date 270 Example: Thu, 19 Dec 85 10:26:48 -0800 271 272 Note that the last two values are locally determined for the maildrop 273 associated with the discussion group and with each message in that 274 maildrop. Note however that the last message in the maildrop have a 275 different MAXIMA and LASTDATE than the discussion group. This often 276 occurs when the maildrop has been archived. 277 278 279 280 281 282Rose [Page 5] 283 284RFC 1082 POP3 Extended Service November 1988 285 286 287 Finally, some local systems provide mechanisms for automatically 288 archiving discussion group mail. In some cases, a two-level archive 289 scheme is used: current mail is kept in the standard maildrop, 290 recent mail is kept in an archive maildrop, and older mail is kept 291 off-line. With this scheme, in addition to having a "standard" 292 maildrop for each discussion group, an "archive" maildrop may also be 293 available. This permits a user agent to examine the most recent 294 archive using the same mechanisms as those used on the current mail. 295 296The XTND Command 297 298 The following commands are valid only in the TRANSACTION state of the 299 POP3. This implies that the POP3 server has already opened the 300 user's maildrop (which may be empty). This maildrop is called the 301 "default maildrop". The phrase "closes the current maildrop" has two 302 meanings, depending on whether the current maildrop is the default 303 maildrop or is a maildrop associated with a discussion group. 304 305 In the former context, when the current maildrop is closed any 306 messages marked as deleted are removed from the maildrop currently in 307 use. The exclusive-access lock on the maildrop is then released 308 along with any implementation-specific resources (e.g., file- 309 descriptors). 310 311 In the latter context, a maildrop associated with a discussion group 312 is considered to be read-only to the POP3 client. In this case, the 313 phrase "closes the current maildrop" merely means that any 314 implementation-specific resources are released. (Hence, the POP3 315 command DELE is a no-op.) 316 317 All the new facilities are introduced via a single POP3 command, 318 XTND. All positive reponses to the XTND command are multi-line. 319 320 The most common multi-line response to the commands contains a 321 "discussion group listing" which presents the name of the discussion 322 group along with it's maxima. In order to simplify parsing all POP3 323 servers are required to use a certain format for discussion group 324 listings: 325 326 NAME SP MAXIMA 327 328 This memo makes no requirement on what follows the maxima in the 329 listing. Minimal implementations should just end that line of the 330 response with a CRLF pair. More advanced implementations may include 331 other information, as parsed from the message. 332 333 NOTE: This memo STRONGLY discourages implementations from 334 supplying additional information in the listing. 335 336 337 338Rose [Page 6] 339 340RFC 1082 POP3 Extended Service November 1988 341 342 343 XTND BBOARDS [name] 344 Arguments: the name of a discussion group (optionally) 345 Restrictions: may only be given in the TRANSACTION state. 346 Discussion: 347 348 If an argument was given, the POP3 server closes the current 349 maildrop. The POP3 server then validates the argument as the name of 350 a discussion group. If this is successful, it opens the maildrop 351 associated with the group, and returns a multi-line response 352 containing the discussion group listing. If the discussion group 353 named is not valid, or the associated archive maildrop is not 354 readable by the user, then an error response is returned. 355 356 If no argument was given, the POP3 server issues a multi-line 357 response. After the initial +OK, for each discussion group known, 358 the POP3 server responds with a line containing the listing for that 359 discussion group. Note that only world-readable discussion groups 360 are included in the multi-line response. 361 362 In order to aid user agents, this memo requires an extension to the 363 scan listing when an "XTND BBOARDS" command has been given. 364 Normally, a scan listing, as generated by the LIST, takes the form: 365 366 MSGNO SIZE 367 368 where MSGNO is the number of the message being listed and SIZE is the 369 size of the message in octets. When reading a maildrop accessed via 370 "XTND BBOARDS", the scan listing takes the form 371 372 MSGNO SIZE MAXIMA 373 374 where MAXIMA is the maxima that was assigned to the message when it 375 was placed in the BBoard. 376 377 Possible Responses: 378 +OK XTND 379 -ERR no such bboard 380 Examples: 381 C: XTND BBOARDS 382 S: +OK XTND 383 S: system 10 384 S: mh-users 100 385 S: . 386 C: XTND BBOARDS system 387 S: + OK XTND 388 S: system 10 389 S: . 390 391 392 393 394Rose [Page 7] 395 396RFC 1082 POP3 Extended Service November 1988 397 398 399 XTND ARCHIVE name 400 Arguments: the name of a discussion group (required) 401 Restrictions: may only be given in the TRANSACTION state. 402 Discussion: 403 404 The POP3 server closes the current maildrop. The POP3 server then 405 validates the argument as the name of a discussion group. If this is 406 successful, it opens the archive maildrop associated with the group, 407 and returns a multi-line response containing the discussion group 408 listing. If the discussion group named is not valid, or the 409 associated archive maildrop is not readable by the user, then an 410 error response is returned. 411 412 In addition, the scan listing generated by the LIST command is 413 augmented (as described above). 414 415 Possible Responses: 416 +OK XTND 417 -ERR no such bboard Examples: 418 C: XTND ARCHIVE system 419 S: + OK XTND 420 S: system 3 421 S: . 422 423 XTND X-BBOARDS name 424 Arguments: the name of a discussion group (required) 425 Restrictions: may only be given in the TRANSACTION state. 426 Discussion: 427 428 The POP3 server validates the argument as the name of a 429 discussion group. If this is unsuccessful, then an error 430 response is returned. Otherwise a multi-line response is 431 returned. The first 14 lines of this response (after the 432 initial +OK) are defined in this memo. Minimal implementations 433 need not include other information (and may omit certain 434 information, outputing a bare CRLF pair). More advanced 435 implementations may include other information. 436 437 Line Information (refer to "Definition of Terms") 438 ---- ----------- 439 1 NAME 440 2 ALIASES, separated by SP 441 3 system-specific: maildrop 442 4 system-specific: archive maildrop 443 5 system-specific: information 444 6 system-specific: maildrop map 445 7 system-specific: encrypted password 446 8 system-specific: local leaders, separated by SP 447 448 449 450Rose [Page 8] 451 452RFC 1082 POP3 Extended Service November 1988 453 454 455 9 ADDRESS 456 10 REQUEST 457 11 system-specific: incoming feed 458 12 system-specific: outgoing feeds 459 13 FLAGS SP MAXIMA 460 14 LASTDATE 461 462 Most of this information is entirely too specific to the UCI Version 463 of the Rand MH Message Handling System [MRose85]. Nevertheless, 464 lines 1, 2, 9, 10, 13, and 14 are of general interest, regardless of 465 the implementation. 466 467 Possible Responses: 468 +OK XTND 469 -ERR no such bboard 470 Examples: 471 C: XTND X-BBOARDS system 472 S: + OK XTND 473 S: system 474 S: local general 475 S: /usr/bboards/system.mbox 476 S: /usr/bboards/archive/system.mbox 477 S: /usr/bboards/.system.cnt 478 S: /usr/bboards/.system.map 479 S: * 480 S: mother 481 S: system@nrtc.northrop.com 482 S: system-request@nrtc.northrop.com 483 S: 484 S: dist-system@nrtc-gremlin.northrop.com 485 S: 01 10 486 S: Thu, 19 Dec 85 00:08:49 -0800 487 S: . 488 489Policy Notes 490 491 Depending on the particular entity administrating the POP3 service 492 host, two additional policies might be implemented: 493 494 1. Private Discussion Groups 495 496 In the general case, discussion groups are world-readable, any user, 497 once logged in (via a terminal, terminal server, or POP3, etc.), is 498 able to read the maildrop for each discussion group known to the POP3 499 service host. Nevertheless, it is desirable, usually for privacy 500 reasons, to implement private discussion groups as well. 501 502 Support of this is consistent with the extensions outlined in this 503 504 505 506Rose [Page 9] 507 508RFC 1082 POP3 Extended Service November 1988 509 510 511 memo. Once the AUTHORIZATION state has successfully concluded, the 512 POP3 server grants the user access to exactly those discussion groups 513 the POP3 service host permits the authenticated user to access. As a 514 "security" feature, discussion groups associated with unreadable 515 maildrops should not be listed in a positive response to the XTND 516 BBOARDS command. 517 518 2. Anonymous POP3 Users 519 520 In order to minimize the authentication problem, a policy permitting 521 "anonymous" access to the world-readable maildrops for discussion 522 groups on the POP3 server may be implemented. 523 524 Support of this is consistent with the extensions outlined in this 525 memo. The POP3 server can be modified to accept a USER command for a 526 well-known pseudonym (i.e., "anonymous") which is valid with any PASS 527 command. As a "security" feature, it is advisable to limit this kind 528 of access to only hosts at the local site, or to hosts named in an 529 access list. 530 531Experiences and Conclusions 532 533 All of the facilities described in this memo and in [RFC1081] have 534 been implemented in MH #6.1. Initial experiences have been, on the 535 whole, very positive. 536 537 After the first implementation, some performance tuning was required. 538 This consisted primarily of caching the datastructures which describe 539 discussion groups in the POP3 server. A second optimization 540 pertained to the client: the program most commonly used to read 541 BBoards in MH was modified to retrieve messages only when needed. 542 Two schemes are used: 543 544 o If only the headers (and the first few lines of the body) of 545 the message are required (e.g., for a scan listing), then only 546 these are retrieved. The resulting output is then cached, on 547 a per-message basis. 548 549 o If the entire message is required, then it is retrieved intact, 550 and cached locally. 551 552 With these optimizations, response time is quite adequate when the 553 POP3 server and client are connected via a high-speed local area 554 network. In fact, the author uses this mechanism to access certain 555 private discussion groups over the Internet. In this case, response 556 is still good. When a 9.6Kbps modem is inserted in the path, 557 response went from good to almost tolerable (fortunately the author 558 only reads a few discussion groups in this fashion). 559 560 561 562Rose [Page 10] 563 564RFC 1082 POP3 Extended Service November 1988 565 566 567 To conclude: the POP3 is a good thing, not only for personal mail but 568 for discussion group mail as well. 569 570 571References 572 573 [RFC1081] Rose, M., "Post Office Protocol - Verison 3 (POP3)", RFC 574 1081, TWG, November 1988. 575 576 [MRose85] Rose, M., and J. Romine, "The Rand MH Message Handling 577 System: User's Manual", University of California, Irvine, 578 November 1985. 579 580 [RFC822] Crocker, D., "Standard for the Format of ARPA-Internet 581 Text Messages", RFC 822, University of Delaware, August 582 1982. 583 584 [RFC918] Reynolds, J., "Post Office Protocol", RFC 918, 585 USC/Information Sciences Institute, October 1984. 586 587 [RFC937] Butler, M., J. Postel, D. Chase, J. Goldberger, and J. 588 Reynolds, "Post Office Protocol - Version 2", RFC 937, 589 USC/Information Sciences Institute, February 1985. 590 591Author's Address: 592 593 594 Marshall Rose 595 The Wollongong Group 596 1129 San Antonio Rd. 597 Palo Alto, California 94303 598 599 Phone: (415) 962-7100 600 601 Email: MRose@TWG.COM 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618Rose [Page 11] 619 620