Reflections on the REST Architectural Style and “Principled Design of the Modern Web Architecture” | Fielding, Taylor, Erenkrantz, Gorlick, Whitehead, Khare, Oreizy

Roy T. Fielding, Richard N. Taylor, Justin Erenkrantz, Michael M. Gorlick, E. James Whitehead, Rohit Khare, Peyman Oreizy; Reflections on the REST Architectural Style and “Principled Design of the Modern Web Architecture; In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017); 2017; pages 4-11 (8 pages); landing.

Performed

Reflections on REST; keynote address; performed at the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017); by one of Roy Fielding, Richard Taylor, Rohit Khare (expect: Rohit Khare); video; 0:47:41; slides (42 slides).

Abstract

Seventeen years after its initial publication at ICSE 2000, the Representational State Transfer (REST) architectural style continues to hold significance as both a guide for understanding how the World Wide Web is designed to work and an example of how principled design, through the application of architectural styles, can impact the development and understanding of large-scale software architecture. However, REST has also become an industry buzzword: frequently abused to suit a particular argument, confused with the general notion of using HTTP, and denigrated for not being more like a programming methodology or implementation framework. In this paper, we chart the history, evolution, and shortcomings of REST, as well as several related architectural styles that it inspired, from the perspective of a chain of doctoral dissertations produced by the University of California’s Institute for Software Research at UC Irvine. These successive theses share a common theme: extending the insights of REST to new domains and, in their own way, exploring the boundary of software engineering as it applies to decentralized software architectures and architectural design. We conclude with discussion of the circumstances, environment, and organizational characteristics that gave rise to this body of work.

Mentions

  • REpresentational State Transfer (REST)
  • Computational REpresentational State Transfer (CREST)
    Computational REST (CREST)
  • Capability Uniform Resource Locator (CURL)
    Capability URL (CURL)
  • COmputAtional State Transfer (COAST)
  • Computing Resource Exchange with Security (COAST)
  • ARRESTED
  • Application Programming Interface (API)
  • Distributed Hash Table (DHT)
  • SIENA (Scalable Internet Event Notification Architectures)
  • XML
  • DHT
  • HTTP
  • REST
  • bit.ly
  • Persistsent Uniform Resource Locator (PURL)
    Persistsent URL (PURL)
  • Notifications
    • e.g. on page transitions
    • HTML ping
    • DOM, onClick, onLoad, onAnything
    • M. Thomson, E. Damaggio, B Raymor. Generic Event Delivery Using HTTP Push. RFC 8030. Internet Engineering Task Force (IETF). 2016.
  • Google Analytics
  • Google Docs
  • Google Sheets
  • AJAX
  • JavaScript
  • HTTP
    • LINK
    • UNLINK
  • Peer-to-Peer (P2P)
  • Decentralized Applications (DAPPs, dApps)
  • Client/Server
  • Web Distributed Authoring and Versioning (WebDAV, WEBDAV)
    • lock-based concurrency control
    • An RPC-based client-server centralized ile system with remote access “over HTTP”
  • Limitations of REST
    • one-shot
    • one-to-one
    • one-way
  • execution engine
  • binding environment
  • COAST
    • Capabilities
      • Services
      • Messaging
      • Interpretation
    • Claims
      • Secure remote code execution (RCE)
      • Live update
      • Novel
      • Monitoring & Traceability
      • Something about refactoring:
        Server abdication, client redelegation, server re-offering (fewer services), client reprogramming of the server.
      • Dynamic Reconfiguration
  • Group Consensus and Simultaneous Agreement (GCSA)
  • WebRTC,
  • Websockets
  • Webhooks
  • HTTP/2
  • Internet of Things (IoT)
  • Content Distrubtion Network (CDN)
  • TrueTime
  • GlobalClock
  • Apache Kafka
  • Amazon Kinesis,
  • Google Cloud Pub/Sub
  • Amazon Lambda,
  • IFTTT
  • ‘assistants’, a natural language conversational product concept, within the buzzy AI business culture. Think: Eliza, that you built in high school.
  • Cassandra
  • NoSQL
  • Federated Learning
  • Merkle Hash Trees (not MHT)
  • Bitcoin
  • <buzz>blockchain</buzz>
  • Git
    • is a decentralized in concept.
    • is not decentrlaized in practice, c.f. GitHub
  • Software-as-a-Service (SaaS)
  • Computational REpresentational State Transfer (CREST)
  • Aura
  • Nikander
  • Trickles
  • network continuations
  • Hypertext Transport Protocol (HTTP)
    • HTTP/1.1
    • HTTP/2
  • DARPA
  • NSF
  • ISR (Irvine Software Rationalization?)
  • Arcadia

Behavior, Asynchrony, State, Execution (BASE)

Concept

Adapability requires the design-time  actions…

LP1
making the parts that are subject to change identifiable, discrete and manipulable.
LP2
providing mechanisms for controlling interactions between the parts subject to change.
LP3
providing techniques for managing state.

Elaborated

  • Peyman Oreizy, Nenad Medvidovic, Richard N. Taylor. Runtime Software Adaptation: Framework, Approaches, and Styles. In Companion of 30th International Conference on Software Engineering (ICSE Companion). 2008. ACM. pages 899–910.
  • Richard N. Taylor, Nenad Medvidovic, Peyman Oreizy. Architectural Styles for Runtime Software Adaptation. In Proceedings of the Eighth Joint Working IEEE/IFIP Conference on Software Architecture and Third European Conference on Software Architecture. IEEE Computer Society, 171–180. 2009.

Exemplars

  • C2
  • CREST
  • MapReduce
  • Pipe-and-Filter
  • Event Notifications
  • “and others.”

Disambiguation

  1. within the transaction formalization of Database Theory
    • Basically Available, Soft state, Eventual consistency (BASE)
      not as used herein.
    • a consistency model wherein everything almost works
      riposte: “eventually we are all dead.”
    • Contra
      • Always Computing In Denial (ACID)
      • Atomicity Consistency Isolation Durability (ACID)
  2. within the Dynamic Software Architectures Theory, page 9.
    • Behavior
    • Asynchrony
    • State
    • Execution
  3. within the ARRESTED Theory, page 10.
    the “mindset” of a node in a distributed network.
    Best-Effort
    Others are making their best effort, as are you.
    Approximate
    There is only approximate knowledge of the state of The Other; your theory of mind is limited & foggy, slacky-latent.
    Self-centered
    Others are self-centered, as are you.
    Efficient
    Make efficient use of the only global resource: communication bandwidth to others; i.e. time is the only finite resource.

Asynchronous, Routed, REpresentational State Transfer with Estimation & Delgation (A+R+REST+E+D, ARRESTED)

  • Polling (and its inverse Asynchrony)
  • Asynchrony (and its inverse Polling)
  • Routing
  • Delegation
  • Estimation

Concept

Theory
REST+P
REST with Polling.
REST+E
REST with Estimation.
A+REST
REST with Asynchrony (callbacks).
R+REST
REST with Routing (packets).
REST+D
REST with Delegation (proxies, gateways).
ARREST
Asynchronous, Routed, REST.
ARREST+E
Asynchronous, Routed, REST, with Estimation.
ARREST+D
Asynchronous, Routed, REST, with Delgation.
ARREST+D
Asynchronous, Routed, REST, with Estimation & Delgation.
ARRESTED
A synonym for slow, yes?
Topology

The metaphor.

Poles
North
Centralized Systems
East
Estimated Systems
South
Decentralized Systems
West
Distributed Systems
Boundaries
now horizon
  • Master-Slave Styles
  • Peer-to-Peer Styles
agency boundary
  • Consensus-Based Styles
  • Consensus-Free Styles

Elaborated

Techniques

  • Bitcoin
  • and other distributed ledger schemes.

Computational REpresentational State Transfer (CREST)

Is just like functional programming.

  • The Poetry
    • mashups of Web culture are “the same as” continuations in programming language theory & culture. c.f. Scheme & SML
    • 300-series redirects are continuations

Principles

CP1
The key abstraction of computation is a resource, named by an URL.
CP2
The representation of a resource is a program, a closure, a continuation, or a binding environment plus metadata to describe the program, closure, continuation, or binding environment.
CP3
All computations are context-free.
CP4
Only a few primitive operations are always available, but additional per-resource operations are also encouraged.
CP5
The presence of intermediaries is promoted.

Concept

  • Ship code+data as a package to evaluate off-box (over there, on their box).
  • Receive code+data as a package to evaluate on-box (here on our box).
  • What could go possibly wrong here? [over there?]

Elaborations

  • Justin R. Erenkrantz. Computational REST: A New Model for Decentralized, Internet-Scale Applications. Ph.D. Dissertation. University of California, Irvine, Irvine, California, USA. 2009.
  • Justin R. Erenkrantz, Michael Gorlick, Girish Suryanarayana, Richard N. Taylor. Harmonizing Architectural Dissonance in REST-based Architectures. Technical Report UCI-ISR-06-18. Institute for Software Research, University of California, Irvine. 2006.
  • Justin R. Erenkrantz, Michael M. Gorlick, Girish Suryanarayana, Richard N. Taylor. From Representations to Computations: The Evolution of Web Architectures. In ACM SIGSOFT Symposium on The Foundations of Software Engineering (FSE). 2007. pages 255–264.
  • Roy T. Fielding. Maintaining distributed hypertext infostructures: Welcome to MOMspider’s Web. In Computer Networks and ISDN Systems, 27, 2. 1994. pages 193–204. doi:10.1016/0169-7552(94)90133-3. Series title? Selected Papers of the First World-Wide Web Conference.

Techniques

  • web mashups
  • session management
  • cookies in client/server interactions
    <quote>, and the (misplaced) role of cookies in client/server interactions</quote>
  • time-dependent resources; e.g. weather forecasts.
  • time-series responses; e.g. stock tickers.

<editorial>Why aren’t cookies necessary again? They uniquely number the consumer base. They are used to develop Measurement, Targeting, Retargeting & Profiling which are the explicit and probably only renumerative use case of the (online) media business model. Oh, right, and paywalls. And, um, public televison-type “membership drive” tip jars.</editorial>

References

There are 59 references.

Abstracted

  • Roy T. Fielding, Richard N. Taylor. Principled Design of the Modern Web Architecture. In Proceedings of the 22nd International Conference on Software Engineering (ICSE). 2000. pages 407–416. IEEE, Limerick, Ireland.

Dissertated

  • Justin R. Erenkrantz. Computational REST: A New Model for Decentralized, Internet-Scale Applications. Ph.D. Dissertation. University of California, Irvine, Irvine, California, USA. 2009.
  • Roy T. Fielding. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. Dissertation. University of California, Irvine, California, USA. 2000.
  • Michael Martin Gorlick. Computational State Transfer: An Architectural Style for Decentralized Systems. Ph.D. Dissertation. Technical Report UCI-ISR-16-3. University of California, Irvine, Irvine, California, USA. 2016.
  • David Alan Halls. Applying Mobile Code to Distributed Systems. Ph.D. Dissertation. University of Cambridge, Cambridge, UK. 1997.
  • Michael Hicks. Dynamic Software Updating. Ph.D. Dissertation. Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania, USA. 2001.
  • Rohit Khare. Extending the REpresentational State Transfer (REST) Architectural Style for Decentralized Systems. Ph.D. Dissertation. University of California, Irvine, California, USA. 2003.
  • Mark Samuel Miller. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. Ph.D. Dissertation. Johns Hopkins University, Baltimore, Maryland, USA. 2006.
  • Peyman Oreizy. Open architecture software: a flexible approach to decentralized software evolution. Ph.D. Dissertation. University of California, Irvine, Irvine, California, USA.
  • Emmet James Whitehead, Jr. An Analysis of the Hypertext Versioning Domain. Ph.D. Dissertation. Univ. of California, Irvine, Irvine, California, USA. 2000.

Complete

  1. T. Aura, P. Niklander. Stateless Connections. In Proceedings of the First International Conference on Information and Communication Security (Lecture Notes In Computer Science), Y. Han, T. Okamoto, S. Qing (editors), Vol. 1334. Springer-Verlag, 1997. pages 87–97.
  2. Tim Berners-Lee, Robert Cailliau, Ari Luotonen, Henrik Frystyk Nielsen, Arthur Secret. The World-Wide Web. In Communications of the ACM, 37, 8. 1994-08. pages 76–82. doi:10.1145/179606.179671.
  3. Tim Berners-Lee, Roy T. Fielding, Larry Masinter. Uniform Resource Identifier (URI): Generic Syntax. RFC 3986. Internet Engineering Task Force (IETF). 2005-01. doi:10.17487/RFC3986.
  4. Tim Berners-Lee, Roy T. Fielding, Henrik Frystyk Nielsen. Hypertext Transfer Protocol – HTTP/1.0. RFC 1945. Internet Engineering Task Force (IETF). 1996-05. doi:10.17487/RFC1945.
  5. Tim Berners-Lee, Jean-Francois Groff. The World Wide Web (a.k.a. WWW). In SIGBIO Newsletter, 12, 3. 1992-09. pages 37–40. doi:10.1145/147126.147133.
  6. Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, Karn Seth. Practical Secure Aggregation for Federated Learning on User-Held Data. In Proceedings of the NIPS Workshop on Private Multi-Party Machine Learning. 2016. landing.
  7. Antonio Carzaniga, David S. Rosenblum, Alexander L. Wolf. Design and Evaluation of a Wide-Area Event Notification Service. In ACM Transactions on Computer Systems, 19, 3. 2001-08. pages 332–383. paywall.
  8. James C. Corbett, Jeffrey Dean et. al. Spanner: Google’s Globally-distributed Database. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI). 2012. pages 251–264. paywall, landing. slides: pptx, event: session.
  9. Chris Dixon. Crypto Tokens: A Breakthrough in Open Network Design. In His Blog, centrally hosted on Medium. 2017-06.
  10. L. Dusseault. HTTP Extensions for Web Distributed Authoring and Versioning (WEBDAV). RFC 4918. Internet Engineering Task Force (IETF). 2007.
  11. Justin R. Erenkrantz. Computational REST: A New Model for Decentralized, Internet-Scale Applications. Ph.D. Dissertation. University of California, Irvine, Irvine, California, USA. 2009.
  12. Justin R. Erenkrantz, Michael Gorlick, Girish Suryanarayana, Richard N. Taylor. Harmonizing Architectural Dissonance in REST-based Architectures. Technical Report UCI-ISR-06-18. Institute for Software Research, University of California, Irvine. 2006.
  13. Justin R. Erenkrantz, Michael M. Gorlick, Girish Suryanarayana, Richard N. Taylor. From Representations to Computations: The Evolution of Web Architectures. In ACM SIGSOFT Symposium on The Foundations of Software Engineering (FSE). 2007. pages 255–264.
  14. Roy T. Fielding. Maintaining distributed hypertext infostructures: Welcome to MOMspider’s Web. In Computer Networks and ISDN Systems, 27, 2. 1994. pages 193–204. doi:10.1016/0169-7552(94)90133-3. Series title? Selected Papers of the First World-Wide Web Conference.
  15. Roy T. Fielding. Relative Uniform Resource Locators. RFC 1808. Internet Engineering Task Force (IETF). 1995-06. doi:10.17487/RFC1808.
  16. Roy T. Fielding. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. Dissertation. University of California, Irvine, California, USA. 2000.
  17. Roy T. Fielding, Gail Kaiser. The Apache HTTP Server Project. In IEEE Internet Computing. 1, 4. 1997-07. pages 88–90. doi:10.1109/4236.612229
  18. Roy T. Fielding, Henrik Frystyk Nielsen, Jeffrey Mogul, Jim Gettys, Tim Berners-Lee. Hypertext Transfer Protocol – HTTP/1.1. RFC 2068. 1997-01. doi:10.17487/RFC2068
  19. Roy T. Fielding, Julian Reschke. Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. RFC 7231. Internet Engineering Task Force (IETF). 2014-06. doi:10.17487/RFC7231.
  20. Roy T. Fielding, Richard N. Taylor. Principled Design of the Modern Web Architecture. In Proceedings of the 22nd International Conference on Software Engineering. 2000. pages 407–416. IEEE, Limerick, Ireland.
  21. Roy T. Fielding, Richard N. Taylor. Principled Design of the Modern Web Architecture. In ACM Transactions on Internet Technology, 2, 2. 2002-05. pages 115–150.
  22. Roy T. Fielding, E. James Whitehead, Jr., Kenneth M. Anderson, Gregory A. Bolcer, Peyman Oreizy, Richard N. Taylor. Web-Based Development of Complex Information Products. In Communications of the ACM, 41, 8. 1998-08. pages 84–92.
  23. Matias Giorgio, Richard N. Taylor. Accountability Through Architecture for Decentralized Systems: A Preliminary Assessment. Technical Report UCI-ISR-15-2. Institute for Software Research, University of California, Irvine. 2015.
  24. Cristiano Giuffrida, Anton Kuijsten, Andrew S. Tanenbaum. 2013. Safe and Automatic Live Update for Operating Systems. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’13). ACM, New York City, New York, USA, 279–292.
  25. Y. Goland, E. Whitehead, A. Faizi, S. Carter, D. Jensen. HTTP Extensions for Distributed Authoring – WEBDAV. RFC 2518. Internet Engineering Task Force. 1999.
  26. Michael Martin Gorlick. Computational State Transfer: An Architectural Style for Decentralized Systems. Ph.D. Dissertation. Technical Report UCI-ISR-16-3. University of California, Irvine, Irvine, California, USA. 2016.
  27. Michael M. Gorlick, Kyle Strasser, Richard N. Taylor. COAST: An Architectural Style for Decentralized On-Demand Tailored Services. In Proceedings of 2012 Joint Working Conference on Software Architecture & 6th European Conference on Software Architecture (WICSA/ECSA). 2012. pages 71–80.
  28. David Alan Halls. Applying Mobile Code to Distributed Systems. Ph.D. Dissertation. University of Cambridge, Cambridge, UK. 1997.
  29. Michael Hicks. Dynamic Software Updating. Ph.D. Dissertation. Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania, USA. 2001.
  30. Irvine Research Unit in Software (IRUS). The Workshop on Internet-Scale Technology (TWIST). A series, 1998-2000.
  31. R. Kadia. Issues Encountered in Building a Flexible Software Development Environment: Lessons from the Arcadia Project. In Proceedings of the Fifth ACM SIGSOFT Symposium on Software Development Environments (SDE). 1992. ACM, New York, NY, USA. pages 169–180. doi:10.1145/142868.143768.
  32. Rohit Khare. Extending the REpresentational State Transfer (REST) Architectural Style for Decentralized Systems. Ph.D. Dissertation. University of California, Irvine, California, USA. 2003.
  33. Rohit Khare, Richard N. Taylor. Extending the REpresentational State Transfer Architectural Style for Decentralized Systems. In Proceedings of the 26th International Conference on Software Engineering (ICSE). 2004. IEEE Computer Society, Edinburgh, Scotland, UK. pages 428–437.
  34. Avinash Lakshman, Prashant Malik. Cassandra: A Decentralized Structured Storage System. In SIGOPS Operating Systems Review, 44, 2. 2010-04. pages 35–40.
  35. David Mazieres. The stellar consensus protocol: A federated model for internet-level consensus. Stellar Development Foundation. 2015.
  36. Mark Samuel Miller. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. Ph.D. Dissertation. Johns Hopkins University, Baltimore, Maryland, USA. 2006.
  37. Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system. 2008.
  38. Peyman Oreizy. Open architecture software: a flexible approach to decentralized software evolution. Ph.D. Dissertation. University of California, Irvine, Irvine, California, USA.
  39. Peyman Oreizy, Michael M. Gorlick, Richard N. Taylor, Dennis Heimbigner, Gregory Johnson, Nenad Medvidovic, Alex Quilici, David Rosenblum. An Architecture-based Approach to Self-Adaptive Software. In IEEE Intelligent Systems, 14, 3. 1999-05 (May-June). pages 54–62.
  40. Peyman Oreizy, Nenad Medvidovic, Richard N. Taylor. Architecture-Based Runtime Software Evolution. In Proceedings of the 20th International Conference on Software Engineering (ICSE). 1998. pages 177–186.
  41. Peyman Oreizy, Nenad Medvidovic, Richard N. Taylor. Runtime Software Adaptation: Framework, Approaches, and Styles. In Companion of 30th International Conference on Software Engineering (ICSE Companion). 2008. ACM. pages 899–910.
  42. Peyman Oreizy, Richard N. Taylor. 1998. On the role of software architectures in runtime system reconfiguration. In IEE Proceedings-Software, 145, 5. 1998. pages 137–145.
  43. Dewayne E. Perry, Alexander L. Wolf. 1992. Foundations for the Study of Software Architecture. In SIGSOFT Software Engineering Notes, 17, 4. 1992-10. pages 40–52. doi:10.1145/141874.141884.
  44. Sean Rhea, Brighten Godfrey, Brad Karp, John Kubiatowicz, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, Harlan Yu. OpenDHT: A Public DHT Service and Its Uses. In SIGCOMM Computing Communication Review, 35, 4. 2005-08. pages 73–84.
  45. Alan Shieh, Andrew C. Myers, Emin G. Sirer. Trickles: A Stateless Network Stack for Improved Scalability, Resilience, and Flexibility. In Proceedings of Symposium on Networked Systems Design and Implementation,/em> (NSDI), Vol. 2. USENIX Association. 2005. pages 175–188.
  46. Alan Shieh, Andrew C. Myers, Emin Gün Sirer. A Stateless Approach to Connection-Oriented Protocols. In ACM Transactions on Computer Systems, 26, 3. 2008-09. pages 8:1–8:50.
  47. James W. Stamos, David K. Gifford. Implementing Remote Evaluation. In IEEE Transactions on Software Engineering, 16, 7. 1990-07. pages 710–722.
  48. James W. Stamos, David K. Gifford. Remote Evaluation. In ACM Transactions on Programming Languages and Systems (TOPLAS), 12, 4. 1990-10. pages 537–564.
  49. Chengzheng Sun, Xiaohua Jia, Yanchun Zhang, Yun Yang, David Chen. Achieving Convergence, Causality Preservation, and Intention Preservation in Real-time Cooperative Editing Systems. In ACM Transactions on Complicating Human Interactions (HCI), 5, 1. 1998-03. pages 63–108.
  50. Richard N. Taylor, Nenad Medvidovic, et al. A Component- and Message- Based Architectural Style for GUI Software. In Transactions on Software Engineering. 1996-06. pages 390–406.
  51. Richard N. Taylor, Nenad Medvidovic, Eric M. Dashofy. Software Architecture: Foundations, Theory, and Practice. John Wiley & Sons. 2010. ASIN:B012AQ8M42: Kindle: no, paper: $151-$600.
  52. Richard N. Taylor, Nenad Medvidovic, Peyman Oreizy. Architectural Styles for Runtime Software Adaptation. In Proceedings of the Eighth Joint Working IEEE/IFIP Conference on Software Architecture and Third European Conference on Software Architecture. IEEE Computer Society, 171–180. 2009.
  53. R.D. Tennant. 1976. The Denotational Semantics of Programming Languages. In Communications of the ACM 19, 8. 1976-08. pages 437–453.
  54. M. Thomson, E. Damaggio, B Raymor. Generic Event Delivery Using HTTP Push. RFC 8030. Internet Engineering Task Force (IETF). 2016.
  55. Emmet James Whitehead, Jr. An Analysis of the Hypertext Versioning Domain. Ph.D. Dissertation. Univ. of California, Irvine, Irvine, California, USA. 2000.
  56. Emmet James Whitehead, Jr., Yaron Goland. The WebDAV Property Design. In Software, Practice and Experience 34 2004, 135–161.
  57. Wikipedia. 2017. Representational state transfer,/a>. In Wikipedia. 2017.
  58. Scott Wolchok, J Alex Halderman. Crawling BitTorrent DHTs for Fun and Profit. In Proceedings of the Fourth USENIX Workshop on Offensive Technologies (WOOT10). 2010.
  59. Gavin Wood. 2014. Ethereum: A secure decentralised generalised transaction ledger. Paper 151. Ethereum Project Yellow Papers 2014.

Previously filled.

Configuring Apache httpd on Fedora 22

Problem

Configure Apache httpd to  (temporarily) run out of a ServerRoot other than the default which is /etc/httpd.

Solution

Create the file /etc/systemd/system/httpd.service containing:

.include /lib/systemd/system/httpd.service
Environment="OPTIONS=-d /local/project/onstage/httpd"

Quotes go around the outside

Referenced

References

Actualities

/etc/sysconfig/httpd

#
# This file is no longer used to configure additional environment variables
# for the httpd process.
#
# It has been replaced by systemd. If you want to customize, the best
# way is to create a file "/etc/systemd/system/httpd.service",
# containing
#   .include /lib/systemd/system/httpd.service
#   ...make your changes here...
# For more info about custom unit files, see How do I set automatic login on a virtual console terminal?

# To set OPTIONS environment variable which was set in this file
# in older versions, you need to create a file named
# "/etc/systemd/system/httpd.service" containing:
#       .include /lib/systemd/system/httpd.service
#       [Service]
#       #
#       # To pass additional options (for instance, -D definitions) to the
#       # httpd binary at startup, set OPTIONS here.
#       #
#       Environment=OPTIONS=-DMY_DEFINE

# Note: With previous versions of httpd, the MPM could be changed by
# editing an "HTTPD" variable here.  With the current version, that
# variable is now ignored.  The MPM is a loadable module, and the
# choice of MPM can be changed by editing the configuration file
# /etc/httpd/conf.modules.d/00-mpm.conf.

Systemd Configuration of Apache httpd

$ find /lib/systemd/system/httpd.s*
/lib/systemd/system/httpd.service
/lib/systemd/system/httpd.service.d
/lib/systemd/system/httpd.socket
/lib/systemd/system/httpd.socket.d

/lib/systemd/system/httpd.service

# It's not recommended to modify this file in-place, because it will be
# overwritten during package upgrades.  If you want to customize, the best
# way is to create a file "/etc/systemd/system/httpd.service",
# containing
#   .include /lib/systemd/system/httpd.service
#   ...make your changes here...
# For more info about custom unit files, see
# http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F

# For example, to pass additional options (for instance, -D definitions) to the
# httpd binary at startup, you need to create a file named
# "/etc/systemd/system/httpd.service" containing:
#	.include /lib/systemd/system/httpd.service
#	[Service]
#	Environment=OPTIONS=-DMY_DEFINE

[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
Environment=LANG=C

ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
# Send SIGWINCH for graceful stop
KillSignal=SIGWINCH
KillMode=mixed
PrivateTmp=true

[Install]
WantedBy=multi-user.target

SOLVED: httpd gives [error] (EAI 2)Name or service not known: Could not resolve host name *s — ignoring!

Indications

$ httpd -t -d `: need a full path ; cd ./opened/httpd; pwd` -f conf/httpd.conf
[thu Jan 01 01:01:01 2015] [error] (EAI 2)Name or service not known: Could not resolve host name *s -- ignoring

Diagnosis

One of the VirtualHost stanzas mentions the hostname *s

Remediation

Fix that. What was intended was * (star)

Actualities

<VirtualHost *s:9999>
SSLEngine On
ServerName www.example.com
DocumentRoot /var/www/html
...etc...
</VirtualHost>

and, as-repaired

Index: com.example.conf
===================================================================
--- com.example.conf    (revision 1234)
+++ com.example.conf    (working copy)
@@ -53,7 +53,7 @@
...etc...
-<VirtualHost *s:9999>
+<VirtualHost *:9999>
SSLEngine Off
ServerName www.example.com
DocumentRoot /var/www/html
...etc...

HOWTO: Use Client-Supplied Certificate Authentication on a WordPress Blog

Client Certificate Authentication; a WordPress Plugin

Theory

 <Location /wp-login.php>
    SSLVerifyClient optional
    <IfModule mod_rewrite.c>
        RewriteEngine   on
        RewriteCond  %{HTTP_USER_AGENT}  .*Safari.*
        RewriteCond  %{SSL:SSL_CLIENT_VERIFY} !=SUCCESS
        RewriteRule  .* /wp-admin [redirect,last]
    </IfModule>
</Location>
<Location /wp-admin>
    SSLVerifyClient require
</Location>

Not Covered

Once the plugin is in place, all interfaces that once used passwords will need to use an appropriate client certificate.  For example the WordPress API.

<Location /xmlrpc.php>
    SSLVerifyClient require
</Location>

Configuration

WordPress

Configuration of WordPress is “standard.”  Get that working first.

Plugin: Client Certificate Authentication

Via: instructions, FAQ.

Apache httpd

/etc/httpd/vhost/com.baker.t99.conf

<VirtualHost *:443>
    SSLEngine On
    SSLCertificateFile      pki/trustanchor/com.baker.STAR.crt
    SSLCertificateKeyFile   pki/trustanchor/com.baker.STAR.key
    SSLCertificateChainFile pki/trustanchor/intermediate.crt
    SSLCACertificateFile    pki/bloggists/all.crt
    SSLCARevocationFile     pki/bloggists/revocation.crl
    # SSLCADNRequestFile (defaults to SSLCACertificateFile)
    # <ifVersion >= 2.4> SSLCARevocationCheck  chain
    ServerName t99.baker.com
    ServerAdmin webmaster@baker.com
    DocumentRoot /var/wordpress/t99
    <Directory "/var/wordpress/t99">
	Options Indexes FollowSymLinks
        # AllowOverride is needed to get wordpress permalinks to work (respect .htaccess)
	AllowOverride All
        # This is needed for permalinks to work on WordPress without using .htaccess files
	RewriteEngine On
        RewriteBase /
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_FILENAME} !-d
	RewriteRule . /index.php [L]
    </Directory>
    RedirectMatch permanent ^/credits?(/|/.*)?$ https://credits.baker.com$1
    RedirectMatch permanent ^/polic(y|ie)s?(/|/.*)?$ https://policies.baker.com$2
    Redirect permanent /robots.txt https://txt-robots.baker.com/
    Header set P3P "CP=\"There is no P3P policy. Learn why here: http://policies.baker.com/p3p\""
    <Location /wp-login.php>
	SSLVerifyClient optional
	RewriteEngine   on
	# SSL:SSL_CLIENT_VERIFY is { NONE, SUCCESS, GENEROUS or FAILED:reason }
	RewriteCond  %{SSL:SSL_CLIENT_VERIFY} !=SUCCESS
	RewriteRule  .* /wp-admin [redirect,last]
    </Location>
    <Location /wp-admin>
	SSLVerifyClient require
        SSLOptions +FakeBasicAuth +ExportCertData
	# The plugin consults
	#   SSL_CLIENT_S_DN_CN for the name.
	#   SSL_CLIENT_S_DN_Email for the email
        SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ and %{SSL_CLIENT_S_DN_O} eq "Baker" and %{SSL_CLIENT_S_DN_OU } in {"Bloggist", "Bloggists"} )
    </Location>
</VirtualHost>

Via: backfill

SOLVED: POST fails in SSL with ‘request body exceeds maximum size (131072) for SSL buffer’

tl;dr

  1. Turn KeepAlive to On
  2. Set SSLRenegBufferSize to a very large value

Condition

  • Large media uploads fail in WordPress
  • Where “large” is over 128K.

Configuration

Diagnostics

The apache log files indicate this problem with lines of the form

request body exceeds maximum size (131072) for SSL buffer, referer: https://example.com/wp-admin/media-new.php
could not buffer message body to allow SSL renegotiation to proceed, referer: https://example.com/wp-admin/media-new.php

Solution

  • Ensure that a GET occurs before the POST such that SSL renegotiation does not occur in the same SSL session.
  • Thus KeepAlive is required..
  • Ensure that the KeepAlive interval is respected between the GET and the POST.
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30

Refereces

Background

Archaeological order … more original materials deeper down the stack…

SOLVED[fail]: Android WebView does not support Client Certificates at all

Problem Statement

Context

Outcome

Partial success…

  • Webware => just works
    • desktop officework browser
      i.e. Firefox 29+
    • mobile browser => “just works”
      i.e. Chrome 40, Blink 537.36, Android 4.4.4 (what is that, Jelly Bean, Key Lime Pie?, Lollipop?)
  • Appware => FAIL
    • Android does not work, cannot be made to work.
    • WordPress Android is unuseable in this mode.

Environment: Webby Officework Desktop

Outcome: just works

Cases

  • Firefox (Fedora) “just works”
  • Chrome (Android) “just works”

Configuration

  • User receives the certificate as a PKCS #12 (a .p12 file)
  • Install fhe certificate
    • … in the browser (Firefox, Linux)
    • … in the operating system (Android)

Environment: WordPress Android

Outcome: FAIL

  • Do not use Android WordPress on these blogs
  • Use the webby interface with Chrome.
  • WordPress Android uses an embedded WebView which does not implement client certificates at all.

Referenced

Others have tried … but Android does not yet support this concept

SOLVED: What causes “Oops, no RSA or DSA server certificate found for ‘servername.tld:0′”

tl;dr => the error message has nothing at all to do with the problem and is horrendously misdirective.

Remediation

  • Disable every single VirtualHost stanza
  • Prove that the server starts with no VirtualHost stanzas at all.
  • Foreach VirtualHoststanza
    • Enable the VirtualHost
    • Restart the server
  • You have identified the VirtualHost stanza containing the problem.

Expectations

  • That SSLEngine On has been omitted from one or more VirtualHost stanzas.
  • That the ServerName or ServerAlias is incorrect or duplicated with another stanza.

References

HOWTO Move a WordPress Blog to a New Domain, refinements to the recipe

Previously: On Moving a WordPress Blog to a New Domain, 2013-01-02.
Especially

On Moving a WordPress Blog to a New Domain

Necessary but not sufficient advice.

What is missing in this recipe is that the graphical content, hyperlinks and banners which are referenced within the blog are not updated to point to the new domain. It seems there are (at least) two cases. The outline here is not fully general but was sufficient for the purposes at hand

Task

Move the blog at admin.emerson.baker.org; to administration.emerson.baker.org

Procedure

  1. Execute the recipe shown above as the admin user.
  2. Clean up the remaining internal pointers the banners in the wp_options table.
  3. Clean up the remaining internal pointers in hyper links within the article content.

Indications

$ rpm -q wordpress mysql
wordpress-3.4.2-2.fc16.noarch
mysql-5.5.28-1.fc16.i686
$ cat /etc/fedora-release
Fedora release 16 (Verne)

Actualities

Substantially what we’re looking to do here is the moral equivalent of this perl code:

s!//admin.emerson.baker.org!//administration.emerson.baker.org!g;

However we need this code executed on every field of every row of every table in the WordPress MySQL database. There does not seem to be an obvious way to do that, so we approximate and guess which tables and which fields need modification.

$ mysql -u wordpress -p
mysql> show tables;
+-----------------------------------+
| Tables_in_org_baker_emerson_admin |
+-----------------------------------+
| wp_commentmeta                    |
| wp_comments                       |
| wp_links                          |
| wp_options                        |
| wp_postmeta                       |
| wp_posts                          |
| wp_term_relationships             |
| wp_term_taxonomy                  |
| wp_terms                          |
| wp_usermeta                       |
| wp_users                          |
+-----------------------------------+
11 rows in set (0.00 sec)

Articles

How bad are the articles?

mysql> select id from wp_posts where post_content like '%//admin.emerson%';
+----+
| id |
+----+
|  4 |
|  5 |
|  6 |
| 13 |
| 16 |
| 20 |
| 27 |
| 42 |
| 79 |
| 81 |
| 82 |
| 83 |
| 84 |
+----+
13 rows in set (0.01 sec)

Remediation

The articles will have to be cleaned up by hand in the WordPress in-browser editor..

Options

The options includes the configurations for the banners and other internal eye candy.

mysql> select option_id, option_name from wp_options where option_value like '%//admin.emerson%';
+-----------+--------------------------------------------------+
| option_id | option_name                                      |
+-----------+--------------------------------------------------+
|       113 | dashboard_widget_options                         |
|       161 | theme_mods_twentyeleven                          |
|       581 | _transient_dash_20494a3d90a6669585674ed0eb8dcd8f |
+-----------+--------------------------------------------------+
3 rows in set (0.01 sec)

mysql> describe wp_options;
+--------------+---------------------+------+-----+---------+----------------+
| Field        | Type                | Null | Key | Default | Extra          |
+--------------+---------------------+------+-----+---------+----------------+
| option_id    | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
| option_name  | varchar(64)         | NO   | UNI |         |                |
| option_value | longtext            | NO   |     | NULL    |                |
| autoload     | varchar(20)         | NO   |     | yes     |                |
+--------------+---------------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

The options are stored as a giant JSON-flavored TLV blob. Watch out for the very very long lines.

mysql> select option_value from wp_options where option_id = 161;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| option_value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| a:3:{s:12:"header_image";s:91:"http://admin.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:17:"header_image_data";O:8:"stdClass":5:{s:13:"attachment_id";i:6;s:3:"url";s:91:"http://admin.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:13:"thumbnail_url";s:91:"http://admin.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:6:"height";i:287;s:5:"width";i:1000;}s:18:"nav_menu_locations";a:1:{s:7:"primary";i:0;}} |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Remediation

Being careful to the string lengths when the string lengths change. In this case, the length of the URLs changes from 91 characters to 100 characters.

mysql> update wp_options set option_value = 'a:3:{s:12:"header_image";s:100:"http://administration.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:17:"header_image_data";O:8:"stdClass":5:{s:13:"attachment_id";i:6;s:3:"url";s:100:"http://administration.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:13:"thumbnail_url";s:100:"http://administration.emerson.baker.org/wp-content/uploads/2013/01/cropped-2832-Emerson-Street-3.png";s:6:"height";i:287;s:5:"width";i:1000;}s:18:"nav_menu_locations";a:1:{s:7:"primary";i:0;}}' where 161 = option_id;
Query OK, 1 row affected (0.53 sec)
Rows matched: 1  Changed: 1  Warnings: 0

dbd-modules | mod_log_dbd, mod_vhost_dbd (Apache CustomLog to an SQL database)

dbd-modules – writing Apache logs straight into a (SQL) database

Mentions

Configuration

The CustomLog format must be a comma-separated list of mod_log_config “%” directives.

CustomLog   logs/access.sql     "%h, %l, %u, %{%Y-%m-%d %H:%M:%S}t, %r, %>s, %b"
DBDLog      logs/access.sql    "INSERT INTO log_table (Host, Rname, User, Tstmp, Request, Status, Bytes) VALUES (%s, %s, %s, %s, %s, %s, %s)"

Via: backfill

Broken

DBDLog can use the name of a statement as its SQL parameter. This name must have been previously defined by a DBDPrepareSQL directive. For example:

LogFormat      "%V, %r"  dbFormat
DBDPrepareSQL  "INSERT INTO log_table (Server, Url) VALUES (%s, %s)"  dbLog
...
CustomLog logs/access.sql  dbFormat
DBDLog    logs/access.sql  dbLog  UseNULLs

This functionality seems to never have been implemented. What the code actually does in this instance is to take the label (dbLog) and attempt to prepare that as a SQL statement on-site. Of course that fails as it isn’t proper SQL. Your /var/log/httpd/error_log will show a critical failure as follows:

[Wed Jan 21 09:38:43.830818 2015] [dbd:error] [pid 11383] (20014)Internal error: AH00632: failed to prepare SQL statements: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'dbLog' at line 1
[Wed Jan 21 09:38:43.831314 2015] [dbd:error] [pid 11383] (20014)Internal error: AH00633: failed to initialise
[Wed Jan 21 09:38:43.831378 2015] [dbd:crit] [pid 11383] (20014)Internal error: AH00636: child init failed!

The workaround is to state the SQL statemen at the site of the DBDLog rather than to try to reference a statement that was prepared by mod_dbd in its DBDPrepareSQL statement.
The code that does not implement this functionality is around line 399 of mod_log_dbd.c.

/* process DBDLog directive */
static const char *setAccessLogQuery(cmd_parms *cmd, void *mconfig, 
                                     const char *name, const char *sql, const char *usenull)
{   
    static long label_num = 0;
    log_dbd_file *file = apr_pcalloc(cmd->pool, sizeof(log_dbd_file));
    log_dbd_svr_conf *conf = 
        (log_dbd_svr_conf *) ap_get_module_config(cmd->server->module_config,
                                              &log_dbd_module);
    if (!dbd_prepare_fn || !dbd_acquire_fn)
        return "mod_dbd must be enabled to use mod_log_dbd";

    if (!log_set_writer_init_fn || !log_set_writer_fn) 
        return "mod_log_config must be enabled to use mod_log_dbd";

    file->stmt_sql = sql;
    file->stmt_label = apr_pstrcat(cmd->pool, "log_dbd_", 
                                   apr_ltoa(cmd->pool, ++label_num), NULL);
    dbd_prepare_fn(cmd->server, sql, file->stmt_label);

    ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, cmd->server,
        "mod_log_dbd: Prepared query (stmt: %s) from: %s",
        file->stmt_label, sql);

    apr_hash_set(conf->files, name, APR_HASH_KEY_STRING, file);

    if (usenull) {
        if(!apr_strnatcasecmp(usenull, "UseNULLs"))
            file->useNull = 1;
        else
             return apr_pstrcat(cmd->pool, "mod_log_dbd: unrecognized option: ",
                                    usenull, NULL);
    }
    return NULL;
}

Building

Instructions

General Recipe

apxs -c mod_vhost_dbd.c
apxs -i mod_vhost_dbd.la
apxs -c mod_log_dbd.c
apxs -i mod_log_dbd.la

Actual Recipe

Executed on Fedora 16, with Apache v2.2.

$ cd /views
$ mkdir com.googlecode.dbd-modules
$ cd com.googlecode.dbd-modules
$ unzip ./dbd-modules-1.0.6.zip 
Archive:  /dbd-modules-1.0.6.zip
  inflating: Makefile.win            
  inflating: mod_log_dbd.c           
  inflating: mod_log_dbd.rc          
  inflating: mod_vhost_dbd.c         
  inflating: mod_vhost_dbd.rc        
  inflating: COPYING                 
  inflating: NOTICE                  
$ apxs -c mod_log_dbd.c
/usr/lib/apr-1/build/libtool --silent --mode=compile gcc -prefer-pic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables  -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -pthread -I/usr/include/httpd  -I/usr/include/apr-1   -I/usr/include/apr-1   -c -o mod_log_dbd.lo mod_log_dbd.c && touch mod_log_dbd.slo
mod_log_dbd.c: In function 'esq':
mod_log_dbd.c:99:5: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
mod_log_dbd.c: In function 'write_log':
mod_log_dbd.c:338:29: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
mod_log_dbd.c:256:23: warning: variable 'rconf' set but not used [-Wunused-but-set-variable]
mod_log_dbd.c: In function 'log_writer_init':
mod_log_dbd.c:367:19: warning: unused variable 'file' [-Wunused-variable]
/usr/lib/apr-1/build/libtool --silent --mode=link gcc -o mod_log_dbd.la  -rpath /usr/lib/httpd/modules -module -avoid-version    mod_log_dbd.lo
$ sudo apxs -i mod_log_dbd.la
/usr/lib/httpd/build/instdso.sh SH_LIBTOOL='/usr/lib/apr-1/build/libtool' mod_log_dbd.la /usr/lib/httpd/modules
/usr/lib/apr-1/build/libtool --mode=install cp mod_log_dbd.la /usr/lib/httpd/modules/
libtool: install: cp .libs/mod_log_dbd.so /usr/lib/httpd/modules/mod_log_dbd.so
libtool: install: cp .libs/mod_log_dbd.lai /usr/lib/httpd/modules/mod_log_dbd.la
libtool: install: cp .libs/mod_log_dbd.a /usr/lib/httpd/modules/mod_log_dbd.a
libtool: install: chmod 644 /usr/lib/httpd/modules/mod_log_dbd.a
libtool: install: ranlib /usr/lib/httpd/modules/mod_log_dbd.a
libtool: finish: PATH="/sbin:/bin:/usr/sbin:/usr/bin:/sbin" ldconfig -n /usr/lib/httpd/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/lib/httpd/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /usr/lib/httpd/modules/mod_log_dbd.so

Update to Firefox 22, and the WordPress text editor arrow keys don’t work (because of Firebug)

Problem Statement

  • The arrow keys and the Home and End keys no longer work in the WordPress wysiwyg text editor.
  • They continue to work in the raw HTML editor.

Severity

  • The wysiwyg text editor is nearly unuseable.
  • The only workaround to move the text entry point is to use the mouse.

Context

  • Fedora 18
  • Firefox 22, updated from Firefox 19
  • There are gobs of other plugins in Firefox: including Firebug.

Diagnosis

  • It’s Firebug’s fault.

Remediation

  • Change the keybindings away from Ctrl+Shift+Arrow
  • The OK vs Cancel does not work in Firebug
    • hit Cancel, it does not Cancel
    • Validate that the changes were accepted via Cancel (which means OK)
  • Restart Firefox (yes, you apparently have to do that)

References

Actualities

Installation and Configuration of the Google+ Crossposting WordPress Plugin

Google+ Crossposting

Plugin Google+ Crossposting

Problem Definition

What Is Wanted

  1. Syndicate (cross-post) your blog posts out onto Google+ for distribution
  2. Use the Google+ identity system to facilitate login on your WordPress blog
    • Tie comments to actual persons.
    • Mitigate spam.
    • Promote clear attribution.

What is Convenient

  1. Syndicate (cross-post) your Public posts on Google+ onto your own venue (WordPress blog).

What You Get

  • The convenient stuff (Google+ has a readonly API).

What You Don’t Get

  • The wanted stuff (Google+ doe does not have a write API).

Other Solutions

Are unsuitable; they require you to disclose your Google password to an external service, into scriptware, or to purchase opaque plugins and services.  They aren’t Google API calls so much as mechanized versions of Google login flows.  Ick.  And vastly dangerous.

  • How to Auto Post to Google Plus from WordPress Blog; In WPSquare; undated (probably late 2012), no date appears on the page.
    • There is no Google API that does this; the Google+ API is readonly
    • The WPGPlus Plugin copies your credentials into the plugin
    • The gPressor, from WarriorForum, by special forum distribution
    • NextScripts has a for-pay addon script; requires their hosting+service arrangements
  • TwooglePlus; used copied user credentials; has been disabled by themselves and by Google

Particulars & Provenances

You need

  • Your Google+ ID (a very long number)
  • A server-to-server Google API Key

Acquiring Your Google+ ID

  1. Go to your Google+ experience (sign in).
  2. Go to your Posts
  3. See the URL which of the form
    https://plus.google.com/NUMBER/posts

Acquiring a Google API Key

  1. Go to https://code.google.com/apis/console/
  2. Read it

Actualities of the key acquisition workflow

Warnings and Gotchas

It is a known effect that the plugin destroys (clears, removes) the Google+ ID and the Google API Key from its settings page if the tokens do not operate without error.  The plugin has no better way of signalling an error than to do this.   So you  need to

  1. Store these items safely some where else as primary storage, not in WordPress
  2. Watch, wait and review to ensure the tokens don’t “disappear”
  3. Realize the connection from Google+ into your blog is shaky at best.

References

Actualities

From acquiring the Google API Key for Google+ access.














The Other Ones

Whereas we became aware after a time that there were others in our land

  • Notes To Self; (socks) Notes to self® socks are high quality athletic performance socks with positive affirmations on the toes and the bottoms of the socks.
  • (the other) Note To Self (blog)
  • Note to Self (movie), 2012; amazon
  • Note to Self (music); (band) From First to Last
  • Note to Self (book); a religious work; amazon
  • Note to Self; The Urban Dictionary
  • Joe’s Notes To Self; (selfhelp) I’m Joe Peacock, and I write myself notes every single day. I don’t know what took me so long but I finally decided to share them. Buy my stupid books, or read …
  • Note to Self; (book) by Andrea Buchanan
    Note To Self a book by Andrea Buchanan. Thirty dynamic women share their inspirational stories on Hardship, Humiliation, Heartbreak and Overcoming it All.

Socks

Hard not to categorize these under self help.

<quote>notes to self® socks target the subconscious mind – most receptive early in the morning and late at night – improving your thoughts, actions and performance!  Made in the U.S.A.<quote>

Blog

Movie

    • Rental $4
    • Note to Self (IMDB), 2012
    • <quote>A “Coming of Age” Film in the Vein of “Love and Basketball” meets “Good Will Hunting”</quote>
    • <quote>Curtis King, a handsome and popular student athlete, (Christian Keyes, Madea Goes to Jail) may know his way around the court, but his heart still needs a game plan. When he decides to keep a journal to give his life a new direction, the path leads him straight to the …</quote>

Music

Band: From First to Last
Era: 2006-10-12
Criticism: loud, blaring, unskilled.  Typical of mid ‘oughties grungoid pop music.

Book

Note to Self: The Discipline of Preaching to Yourself (Re: Lit Books)

Joe Thorn; Note to Self: The Discipline of Preaching to Yourself; 2011-04-07.

A work of the Christian religion.

On the use of <span style=”display:none;”> in WordPress comment spam

The existence of comment spam in WordPress makes the whole enterprise of an open loop commentariat problematic (which is delicate indirect fancyspeak for: the advice is to turn off comments).

Here’s a dump of the technique for using <span style=”display:none;”> in the source blog to spam up other blogs to drive traffic back.

Background

  • (factually) on 2012-12-26 the publication of Venkat Rao; ribbonfarm; The Crucible Effect and the Scarcity of Collective Attention occurred in Backfill of ‘Note to Self’.  Factually, because I represent that I did it.
  • (apparently) 2012-12-10, the publication 1000 Raving Fans – Part Two occurred in the blog Work With David Wood which appears to be promoting techniques for Multi-Level Marketing (MLM) on Facebook.  Apparently, because that’s what the dates on the pages say; and we know that WordPress blog posts are easily backdated.
  • (actually) on 2013-01-12 a comment pingback appears at Backfill indicating that the Rao/ribbonfarm/Crucible+Attention post was referenced.  Yet there is no mention of the article in the source blog and the WordPress provenance of the comment are incomprehensible, being raw HTML and JavaScript at the code level.
  • (result) on 2013-01-12 the comment was declared to be spam.

Technique

The technique involves using <span style="display:none;"> within the offending referencing article somewhere. This acquires a comment indicator back in the victim source blog but does not offer any material visibility in the offending referencing blog. The HTML code on the offending referencing blog is:

<a href="http://backfill.note-to-self.baker.com/2012/12/26/venkat-rao-ribbonfarm-the-crucible-effect-and-the-scarcity-of-collective-attention/" rel="nofollow"><span style="display:none;">Venkat Rao; ribbonfarm; The Crucible Effect and the Scarcity of Collective Attention</span></a>

which presented in readable form is:

<a href=”http://backfill.note-to-self.baker.com/2012/12/26/venkat-rao-ribbonfarm-the-crucible-effect-and-the-scarcity-of-collective-attention/”
rel=”nofollow”>
<span style=”display:none;”>
Venkat Rao; ribbonfarm; The Crucible Effect and the Scarcity of Collective Attention
</span>
</a>

Comment Approval User Experience

The comment approval user experience does not offer much in the way of support to detect this condition. The snippet given for the comment approval workflow renders raw HTML and JavaScript code.

Actualities

Use view images on the images to get the high resolution images; the link targets is the actual offending blog post itself.

Many many other blogs are referenced with this technique.  To wit:

<a href="http://www.empowernetwork.com/giudiced/blog/1000-raving-fans/" rel="nofollow"><span style="display:none;">1000 Raving Fans</span></a><a href="http://backfill.note-to-self.baker.com/2012/12/26/venkat-rao-ribbonfarm-the-crucible-effect-and-the-scarcity-of-collective-attention/" rel="nofollow"><span style="display:none;">Venkat Rao; ribbonfarm; The Crucible Effect and the Scarcity of Collective Attention</span></a><a href="http://glowdarkshoes.edublogs.org/2013/01/05/list-building-providing-a-free-giveaway-to-turn-your-prospects-into-clients-and-raving-fans-jordan-high-heels/" rel="nofollow"><span style="display:none;">List Building &#8211; Providing a Free Giveaway to Turn Your Prospects Into Clients and Raving Fans &#8211; Jordan High Heels</span></a><a href="http://hoffmanmurphyteam.wordpress.com/2013/01/10/thank-you-for-all-your-efforts-a-raving-fans-testimonial/" rel="nofollow"><span style="display:none;">Thank You For All Your Efforts – A Raving Fans Testimonial</span></a><a href="http://workatshindigz.wordpress.com/2013/01/10/shindigz-internet-marketing-looking-to-expand/" rel="nofollow"><span style="display:none;">Shindigz Internet Marketing Team Looking to Expand</span></a><a href="http://www2.webmasterradio.fm/affiliate-buzz/2013/01/10/affiliate-summit-west-2013-preview-social-media-wing-of-school-of-internet-marketing" rel="nofollow"><span style="display:none;">Affiliate Summit West 2013 Preview; Social Media Wing of School of Internet Marketing</span></a><a href="http://photographybusiness.wordpress.com/2013/01/10/9-things-to-do-to-drive-your-photography-customers-crazy/" rel="nofollow"><span style="display:none;">9 Things To Do To Drive Your Photography Customers Crazy</span></a><a href="http://www.opace.co.uk/blog/happy-birthday-text-how-you-changed-internet-marketing-and-social-media" rel="nofollow"><span style="display:none;">Happy Birthday, text! How texting changed internet marketing and social media</span></a><a href="http://www.revolutionincome.com/farewell-to-one-of-my-original-mentors-zig-ziglar-november-28-2012/" rel="nofollow"><span style="display:none;">Farewell To One Of My Original Mentors - Zig Ziglar - November 28, 2012 - Revolution Income</span></a><a href="http://beautiful931.hereshoppingonline.com/review-for-harmony-gelish-top-5oz-base-5oz-set-of-2-high-quality-products-ship-now-get-now/" rel="nofollow"><span style="display:none;">Review for Harmony Gelish Top .5oz Base .5oz &#8220;Set of 2&#8243; High Quality Products. Ship Now Get Now</span></a><link rel="stylesheet" type="text/css" href="http://workwithdavidwood.com/wp-content/plugins/getsocial/lib/getsocialstyles.php?color=EAEAF4&border=D9D9ED&w=73&strip=dark&prehide=no&rc=yes&stralign=right" />

Bring up WordPress on Fedora 17

Design

  • apache httpd runs as user apache:apache
  • /var/wordpress/NEWBLOG contains the blog
  • /etc/wordpress/NEWBLOG.php contains the blog configuration (wp-config.php)
  • /var/http/VIRTHOST contains other virtual hosts served by apache httpd
  • /var/www contains the original apache httpd content area (with its SELinux labels)

SELinux Considerations

  • You will have issues as you are installing in nonstandard places.
  • See the recipe herein.
  • Stick with it, you want the protection

Known Problems

Bug 891764 php-simplepie 1.3.1 breaks WordPress

Install Packages

n.b. this may not be the minimal set

  • (sudo) yum install -y mysql-libs mysql mysql-server >& o.yum_install.out
  • (sudo) yum install -y php-{cli,xml,gd,IDNA_Convert,soap,pdo,mysql,simplepie,common} >& o.yum_install.out
  • (sudo) yum install -y wordpress wordpress-plugin-defaults wordpress-plugin-bad-behavior >& o.yum_install.out

Bring up MySQL

  • (sudo) systemctl enable mysqld.service
  • (sudo) systemctl start mysqld.service
  • mysqladmin -u root password $uuid1
  • mysql -u root -p
    • supply password $uuid1
    • create user wordpress identified by '$uuid2';
    • select password('$uuid2');
    • create database NEWBLOG;
    • grant all privileges on NEWBLOG.* to wordpress@localhost identified by password 'hashed-uuid2';
    • quit

Install & Configure WordPress

  • (sudo) mkdir /var/wordpress
  • cd /var/wordpress
  • (sudo) cp -rpc /usr/share/wordpress NEWBLOG/.
    • -r is recursive
    • -p is preserve permissions
    • -c is preserve SELinux context labels
  • (sudo) chown -R apache:apache NEWBLOG/.
  • cd NEWBLOG
  • (sudo) rm wp-config.php
  • (sudo) ln -s ../../../etc/wordpress/NEWBLOG.php wp-config.php
  • (sudo) vi /etc/wordpress/NEWBLOG.php

Hack the Permissions

Permissions on /etc/wordpress

$ ls -la /etc/wordpress
total 24
drwxr-xr-x.   2 root root  4096 Jan 10 16:15 .
drwxr-xr-x. 137 root root 12288 Jan 10 15:01 ..
-rw-r--r--.   1 root root  3178 Jan 10 16:15 NEWBLOG.php
-rw-r--r--.   1 root root  3177 Dec 12 05:55 wp-config.php

$ sudo chown -R apache:apache /etc/wordpress

$ sudo chmod o-rx -R /etc/wordpress 

$ ls -la /etc/wordpress
ls: cannot open directory /etc/wordpress: Permission denied

$ sudo ls -la /etc/wordpress
total 24
drwxr-x---.   2 apache apache  4096 Jan 10 16:15 .
drwxr-xr-x. 137 root   root   12288 Jan 10 15:01 ..
-rw-r-----.   1 apache apache  3178 Jan 10 16:15 NEWBLOG.php
-rw-r-----.   1 apache apache  3177 Dec 12 05:55 wp-config.php

SELinux Labels on /var/wordpress/NEWBLOG

The following label patterns need to be available

semanage -i - <

Then the tree needs to be “relabelable” so that in case a restorecon action happens, the system isn’t broken. So in concept:

  • copy_context “NEWBLOG
  • semanage_patterns “NEWBLOG
  • relabel_tree “NEWBLOG

The packaged script is fixup. To use

    1. (sudo) mkdir /var/wordpress/selinux
    2. download it to /var/wordpress/selinux/fixup
    3. (sudo) chmod a+x /var/wordpress/selinux/fixup
    4. cd /var/wordpress
    5. (sudo) selinux/fixup NEWBLOG

Once done, see

      • /var/wordpress/NEWBLOG/o.chcon.out
      • /var/wordpress/NEWBLOG/o.semanage.out
      • /var/wordpress/NEWBLOG/o.restorecon.out

Once done, inspect the SELinux label patterns:

$ sudo semanage -o -
boolean -D
login -D
login -a -s unconfined_u -r 's0-s0:c0.c1023' __default__
login -a -s unconfined_u -r 's0-s0:c0.c1023' root
login -a -s system_u -r 's0-s0:c0.c1023' system_u
user -D
port -D
interface -D
node -D
fcontext -D
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/.htaccess'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/blogs.dir(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/cache(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/plugins(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/themes(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/upgrade(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/uploads(/.*)?'
fcontext -a -f 'all files' -t httpd_sys_rw_content_t '/var/wordpress/NEWBLOG/wp-content/wp-cache-config.php'

Configuring Apache httpd

In /etc/httpd/conf/httpd.conf is the declaration of name-based virtual hosting:

NameVirtualHost *:80
Include vhost/*.conf
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    ServerName _default_
    ServerAlias *
    ErrorLog logs/error_log
    CustomLog logs/access_log common
</VirtualHost>

The NEWBLOG entry is constructed with a ServerAlias such that it will always match, and since it appears “First” in the ordering of the VirtualHost declarations, then it will be the only match. The default will never match. And in /etc/httpd/vhost/NEWBLOG.conf is the declaration:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/wordpress/NEWBLOG
    ServerName _dummy_
    ServerAlias *
    ErrorLog logs/NEWBLOG/error_log
    CustomLog logs/NEWBLOG/access_log common
</VirtualHost>

Be sure to create the log file directory

  • (sudo) mkdir /var/log/httpd/NEWBLOG

(re)Start the server

$ sudo systemctl restart httpd.service

Grand Unification Theory | redux and a tour down memory lane

Upon the occasion of the reading of the logfiles

==> /var/log/httpd/com.baker.www/error_log <==
[Wed Jan 1 23:02:46 2013] [error] [client 2600:3c01:e001:0d00::/60] File does not exist: /var/http/com.baker.www/html/grand-unification-theory

They mostly come at nightmostly ..

The Bots.  They come for http://www.baker.com/grand-unification-theory

Some history is in order…  Once upon a time there was (what we now know as) a blog and a distribution list called Grand Unification Theory.  Way back, back before Note to Self, before Backfill, before Landfill, which you can’t get at. Mutatis mutandis. Ceteris paribus. In a time long ago and far away, so long ago and so far away, in fact, that it was before the Internet even existed. This was before time itself. There was the granddaddy of them all … Grand Unification Theory with a Boneyard, The Yard and Back 40 (Acres) and Archives running back into mid-year 1990. A general walk down memory lane of baker.com from 1998. A decade and a half ago. What a time it was!

On Moving a WordPress Blog to a New Domain

Activity

Concern

  • Is there going to be a problem with one of the blogs being in a subdomain of the other one?
  • The whole system works on HTML4 cookies which are on at a domain-superdomain basis.

References

Recipe #1

This may be enough … maybe, but see Recipe #2

From Changing The Site URL

  1. Edit the wp-config.php file.
  2. After the define statements (just before the comment line that says “That’s all, stop editing!”), insert a new line, and type: define('RELOCATE',true);
  3. Save your wp-config.php file.
  4. Open a web browser and manually point it to wp-login.php on the new URL.
    e.g. http://backfill.note-to-self.baker.com/wp-login.php
  5. Login as per normal.
  6. Verify you are at the correct URL.
  7. Login as Admininstrator; e.g. the user admin.
  8. As Administrator, navigate to Settings > General and verify that both the URL settings are correct.
  9. Hit Save Changes.
  10. Once this has been fixed, edit wp-config.php and either completely remove the line that you added (delete the whole line), comment it out (with //) or change the true value to false if you think it’s likely you will be relocating again.

Actualities of Recipe #1

The fragment of edit wp-config.php modified in place.

/**
 * For developers: WordPress debugging mode.
 *
 * Change this to true to enable the display of notices during development.
 * It is strongly recommended that plugin and theme developers use WP_DEBUG
 * in their development environments.
 */
define('WP_DEBUG', true);

define('RELOCATE', false);

/* That's all, stop editing! Happy blogging. */

/** Absolute path to the WordPress directory. */
if ( !defined('ABSPATH') )
	define('ABSPATH', dirname(__FILE__) . '/');

/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');

The WordPress Administration UI at Settings > General

Administrator login at Settings>General showing the update of the WordPress URL and Site URL to the new URL

Recipe #2

The document writers distinguish the various cases in terms of gauzy abstractions which are somewhat hard to grapple pin down with precision in the case of virtual hosting and DNS CNAME trickery.

  • Changing to a different server.
  • Changing the site URL.
  • Moving the WordPress files from one location to another
  • Moving from on one server to another location on another server

Various levels of intervention are required for each of these. It’s not at all clear that they are disjoint cases. For example, see When Your Domain Name or URLs Change within Moving WordPress).

The cases seem to be

  1. When the domain name changes
    e.g. backfill.baker.com becomes ensconced as backfill.note-to-self.baker.com
  2. When the path component changes; e.g.
    www.emerson.baker.org/blogs grows up and moves out to blogs.emerson.baker.org

Continuing

Separately, prove that the old URL is no longer referenced in any content or other files of the site.

$ cd backfill
$ grep -e backfill.baker.com -rI .
no output

No output is good output.

Apache Virtual Hosting Backstory

This virtual hosting stanza must remain in force within httpd.conf … forever.

<VirtualHost *:80>
    ServerName backfill.baker.com
    ServerAdmin webmaster@baker.org
    Redirect permanent / http://backfill.note-to-self.baker.com/
</VirtualHost>

DNS CNAME Backstory

This CNAME must remain in force within DNS … forever.

backfill.baker.com.	CNAME	backfill.note-to-self.baker.com.

On the appearance of Pingbacks in WordPress article comments

In the note TiVo Connections and Service Ports of 2012-12-19 we see the pingback appear as Pingback: ryrove List which references http://jkproveclub.com/ as follows:

This appears as follows in the WordPress HTML code

<div id="comments">
    <h2 id="comments-title">
        One thought on “<span>TiVo Connections and Service Ports</span>”
    </h2>
    <ol class="commentlist">
        <li class="post pingback">
            <p>Pingback: <a href='http://jkproveclub.com' rel='external nofollow' class='url'>ryrove List</a></p>
        </li>
    </ol>
</div>

The confusing part for an administrator is that jkproveclub doesn’t exist in DNS or in the search indices (e.g. in a G. search); neither does ryrove (search).  This isn’t exactly spam because it’s flattering and doesn’t contain spammy qualities.  Yet it is unhelpful in its terseness and opacity.

The comment dashboard in WordPress is also unhelpful.

Of note here

  • The IPv6 address is the address of my proxy and is already known to me.
  • There is no user ryrove registered in the Note To Self WordPress user database.
  • The comment text does not appear on this blog (my blog) at all; it only seems to appear in the approval dashboard.
    Wow, awesome blog format! How long have you been blogging for?

References

 

Bring up multiple WordPress instances on Fedora 16

Problem Statement

  • WordPress, circa WordPress 3.4.2
  • Fedora 16
    • SELinux to be used
    • Dual-stack visibility (IPv4 and IPv6)3
  • Multitenancy => multiple nonstandard locations for the code, config & data management trees.
    • Multiple blog sites
    • Multiple web sites

Design

  • /etc/httpd virtual hosting
  • /var/http contains multiple web sites at /var/http/sitename
  • /var/wordpress contains multiple blog sites at /var/wordpress/blogname
  • /etc/wordpress contains the multiple wp-config.php, each named sitename.php
    Therefore the symlink /var/wordpress/sitename/wp-config.php has the (pointer) value ../../../etc/wordpress/sitename.php

Expectations

Expect policy-version and release dependencies surrounding the SELinux in the new location. Fedora 16 isn’t the latest thing out there, so some of the policy problems will already have been fixed. Other policy problems will appear do the new use case: installing in an unexpected location.

Concept: SELinux is good, and true and wonderful, and it will protect you. Use it, learn it, live it. The pain you feel is operations safety entering your system.

Rewrite

Redirect

Previous Experiences

cd /var/wordpress/landfill
sudo chcon -v -R --reference=/var/www/html .
sudo semanage fcontext -a -t httpd_sys_rw_content_t
/var/wordpress/landfill/wp-content/uploads
sudo restorecon -v /var/wordpress/landfill/wp-content/uploads
getsebool -a
sudo setsebool -P httpd_can_network_connect on

Outstanding Issues

  1. Uploading a new header; error message: Image could not be processed. Please go back and try again.
    • nothing in /var/www/httpd/landfill/error_log
    • nothing in /var/log/messages/

    Suggestion: yum install php-gd
    Reference: cite
    Result: not helpful (no change)

    Dec 1 00:07:28 opened yum[13986]: Installed: t1lib-5.1.2-9.fc16.i686
    Dec 1 00:07:28 opened yum[13986]: Installed: php-gd-5.3.18-1.fc16.i686

References