ref/rep/partition.html

<!--$Id: partition.so,v 1.8 2007/10/03 17:48:44 sue Exp $-->
<!--Copyright (c) 1997,2008 Oracle.  All rights reserved.-->
<!--See the file LICENSE for redistribution information.-->
<html>
<head>
<title>Berkeley DB Reference Guide: Network partitions</title>
<meta name="description" content="Berkeley DB: An embedded database programmatic toolkit.">
<meta name="keywords" content="embedded,database,programmatic,toolkit,btree,hash,hashing,transaction,transactions,locking,logging,access method,access methods,Java,C,C++">
</head>
<body bgcolor=white>
<table width="100%"><tr valign=top>
<td><b><dl><dt>Berkeley DB Reference Guide:<dd>Berkeley DB Replication</dl></b></td>
<td align=right><a href="/rep/clock_skew.html"><img src="/images/prev.gif" alt="Prev"></a><a href="/toc.html"><img src="/images/ref.gif" alt="Ref"></a><a href="/rep/faq.html"><img src="/images/next.gif" alt="Next"></a>
</td></tr></table>
<p align=center><b>Network partitions</b></p>
<p>The Berkeley DB replication implementation can be affected by network
partitioning problems.</p>
<p>For example, consider a replication group with N members.  The network
partitions with the master on one side and more than N/2 of the sites
on the other side.  The sites on the side with the master will continue
forward, and the master will continue to accept write queries for the
databases.  Unfortunately, the sites on the other side of the partition,
realizing they no longer have a master, will hold an election.  The
election will succeed as there are more than N/2 of the total sites
participating, and there will then be two masters for the replication
group.  Since both masters are potentially accepting write queries, the
databases could diverge in incompatible ways.</p>
<p>If multiple masters are ever found to exist in a replication group, a
master detecting the problem will return <a href="/api_c/rep_message.html#DB_REP_DUPMASTER">DB_REP_DUPMASTER</a>.  If
the application sees this return, it should reconfigure itself as a
client (by calling <a href="/api_c/rep_start.html">DB_ENV-&gt;rep_start</a>), and then call for an election
(by calling <a href="/api_c/rep_elect.html">DB_ENV-&gt;rep_elect</a>).  The site that wins the election may be
one of the two previous masters, or it may be another site entirely.
Regardless, the winning system will bring all of the other systems into
conformance.</p>
<p>As another example, consider a replication group with a master
environment and two clients A and B, where client A may upgrade to
master status and client B cannot.  Then, assume client A is partitioned
from the other two database environments, and it becomes out-of-date
with respect to the master.  Then, assume the master crashes and does
not come back on-line.  Subsequently, the network partition is restored,
and clients A and B hold an election.  As client B cannot win the
election, client A will win by default, and in order to get back into
sync with client B, possibly committed transactions on client B will be
unrolled until the two sites can once again move forward together.</p>
<p>In both of these examples, there is a phase where a newly elected master
brings the members of a replication group into conformance with itself
so that it can start sending new information to them.  This can result
in the loss of information as previously committed transactions are
unrolled.</p>
<p>In architectures where network partitions are an issue, applications
may want to implement a heart-beat protocol to minimize the consequences
of a bad network partition.  As long as a master is able to contact at
least half of the sites in the replication group, it is impossible for
there to be two masters.  If the master can no longer contact a
sufficient number of systems, it should reconfigure itself as a client,
and hold an election.  Replication Manager does not currently
implement such a feature, so this technique is only available to
applications which use the Base replication API.</p>
<p>There is another tool applications can use to minimize the damage in
the case of a network partition.  By specifying an <b>nsites</b>
argument to <a href="/api_c/rep_elect.html">DB_ENV-&gt;rep_elect</a> that is larger than the actual number of
database environments in the replication group, applications can keep
systems from declaring themselves the master unless they can talk to
a large percentage of the sites in the system.  For example, if there
are 20 database environments in the replication group, and an argument
of 30 is specified to the <a href="/api_c/rep_elect.html">DB_ENV-&gt;rep_elect</a> method, then a system will have
to be able to talk to at least 16 of the sites to declare itself the
master.</p>
<p>Replication Manager uses the value of <b>nsites</b> (configured by
the <a href="/api_c/rep_nsites.html">DB_ENV-&gt;rep_set_nsites</a> method) for elections as well as in calculating how
many acknowledgements to wait for when sending a
<a href="/api_c/rep_transport.html#DB_REP_PERMANENT">DB_REP_PERMANENT</a> message.  So this technique may be useful here
as well, unless the application uses the <a href="/api_c/repmgr_ack_policy.html#DB_REPMGR_ACKS_ALL">DB_REPMGR_ACKS_ALL</a> or
<a href="/api_c/repmgr_ack_policy.html#DB_REPMGR_ACKS_ALL_PEERS">DB_REPMGR_ACKS_ALL_PEERS</a> acknowledgement policies.</p>
<p>Specifying a <b>nsites</b> argument to <a href="/api_c/rep_elect.html">DB_ENV-&gt;rep_elect</a> that is
smaller than the actual number of database environments in the
replication group has its uses as well.  For example, consider a
replication group with 2 environments.  If they are partitioned from
each other, neither of the sites could ever get enough votes to become
the master.  A reasonable alternative would be to specify a
<b>nsites</b> argument of 2 to one of the systems and a <b>nsites</b>
argument of 1 to the other.  That way, one of the systems could win
elections even when partitioned, while the other one could not.  This
would allow one of the systems to continue accepting write
queries after the partition.</p>
<p>In a 2-site group, Replication Manager reacts to the loss of
communication with the master by assuming the master has crashed: the
surviving client simply declares itself to be master.  Thus it avoids
the problem of the survivor never being able to get enough votes to
prevail.  But it does leave the group vulnerable to the risk of
multiple masters, if both sites are running but cannot communicate.</p>
<p>These scenarios stress the importance of good network infrastructure in
Berkeley DB replicated environments.  When replicating database environments
over sufficiently lossy networking, the best solution may well be to
pick a single master, and only hold elections when human intervention
has determined the selected master is unable to recover at all.</p>
<table width="100%"><tr><td><br></td><td align=right><a href="/rep/clock_skew.html"><img src="/images/prev.gif" alt="Prev"></a><a href="/toc.html"><img src="/images/ref.gif" alt="Ref"></a><a href="/rep/faq.html"><img src="/images/next.gif" alt="Next"></a>
</td></tr></table>
<p><font size=1>Copyright (c) 1996,2008 Oracle.  All rights reserved.</font>
</body>
</html>