Lines Matching refs:packet
3 * SSH packet transport layer.
53 * The code below employs two main data structures: The packet queue,
57 * Shared ownership of a packet is controlled via reference counting. Inside
58 * the transport system are a total of five packet owners:
60 * - the packet queue,
66 * Normal operation is as follows: The initial reference of the packet is
67 * obtained by submitting the packet and queuing it. The receiver thread takes
69 * but takes over the reference (removing it from the queue). If the packet is
71 * sets-up the timeout and adds the packet to the pending set before starting
74 * by the transmitter thread is dropped. If the packet is unsequenced (i.e.
75 * does not need an ACK), the packet is completed by the transmitter thread
79 * reference to the packet from the pending set. The receiver thread will then
80 * complete the packet and drop its reference.
86 * scheduled depending on the earliest packet timeout expiration date,
88 * timeout of a packet has expired, it is re-submitted and the number of tries
89 * of this packet is incremented. If this number reaches its limit, the packet
92 * On transmission failure (such as repeated packet timeouts), the completion
95 * To ensure that a packet eventually leaves the system it is marked as
97 * canceled. Marking a packet as "locked" has the effect that passing and
98 * creating new references of the packet is disallowed. This means that the
99 * packet cannot be added to the queue, the pending set, and the timeout, or
101 * packet from the system it has to be marked as locked and subsequently all
106 * Note that the packet completion callback is, in case of success and for a
107 * sequenced packet, guaranteed to run on the receiver thread, thus providing
108 * a way to reliably identify responses to the packet. The packet completion
109 * callback is only run once and it does not indicate that the packet has
111 * triggered when the reference count of the packet reaches zero). In case of
113 * the packet is being re-transmitted while the completion callback runs.
115 * the packet is canceled.
119 * Flags are used to indicate the state and progression of a packet. Some flags
123 * Indicates if the packet is locked. If the packet is locked, passing and/or
124 * creating additional references to the packet is forbidden. The packet thus
126 * that the packet state flags may still change (e.g. it may be marked as
130 * Indicates if the packet completion callback has been executed or is about
131 * to be executed. This flag is used to ensure that the packet completion
135 * Indicates if a packet is present in the submission queue or not. This flag
137 * presence of the packet in the queue.
140 * Indicates if a packet is present in the set of pending packets or not.
142 * coherent to the presence of the packet in the pending set.
145 * Indicates if the packet is currently transmitting. In case of
151 * Indicates if the packet has been transmitted. This flag is not cleared by
155 * Indicates if the packet has been acknowledged by the client. There are no
156 * other guarantees given. For example, the packet may still be canceled
161 * Indicates if the packet has been canceled from the outside. There are no
162 * other guarantees given. Specifically, the packet may be completed by
170 * - The packet priority must be accessed only while holding the queue lock.
172 * - The packet timestamp must be accessed only while holding the pending
177 * SSH_PTL_MAX_PACKET_TRIES - Maximum transmission attempts for packet.
179 * Maximum number of transmission attempts per sequenced packet in case of
180 * time-outs. Must be smaller than 16. If the packet times out after this
181 * amount of tries, the packet will be completed with %-ETIMEDOUT as status
189 * Timeout in jiffies for packet transmission via the underlying serial
190 * device. If transmitting the packet takes longer than this timeout, the
191 * packet will be completed with -ETIMEDOUT. It will not be re-submitted.
199 * time-frame after starting transmission, the packet will be re-submitted.
238 * re-transmits the packet-to-be-ACKed and the driver should detect it as
240 * for the re-transmitted packet.
263 * data packet.
265 * Useful to test re-transmit timeout of the driver. If the data packet has not
266 * been ACKed after a certain time, the driver should re-transmit the packet up
292 * Causes the packet data to be actively corrupted by overwriting it with
294 * with a NAK packet. Useful to test handling of NAK packets received by the
329 static bool __ssh_ptl_should_drop_ack_packet(struct ssh_packet *packet)
334 trace_ssam_ei_tx_drop_ack_packet(packet);
335 ptl_info(packet->ptl, "packet error injection: dropping ACK packet %p\n",
336 packet);
341 static bool __ssh_ptl_should_drop_nak_packet(struct ssh_packet *packet)
346 trace_ssam_ei_tx_drop_nak_packet(packet);
347 ptl_info(packet->ptl, "packet error injection: dropping NAK packet %p\n",
348 packet);
353 static bool __ssh_ptl_should_drop_dsq_packet(struct ssh_packet *packet)
358 trace_ssam_ei_tx_drop_dsq_packet(packet);
359 ptl_info(packet->ptl,
360 "packet error injection: dropping sequenced data packet %p\n",
361 packet);
366 static bool ssh_ptl_should_drop_packet(struct ssh_packet *packet)
369 if (!packet->data.ptr || !packet->data.len)
372 switch (packet->data.ptr[SSH_MSGOFFSET_FRAME(type)]) {
374 return __ssh_ptl_should_drop_ack_packet(packet);
377 return __ssh_ptl_should_drop_nak_packet(packet);
380 return __ssh_ptl_should_drop_dsq_packet(packet);
387 static int ssh_ptl_write_buf(struct ssh_ptl *ptl, struct ssh_packet *packet,
394 trace_ssam_ei_tx_fail_write(packet, status);
395 ptl_info(packet->ptl,
396 "packet error injection: simulating transmit error %d, packet %p\n",
397 status, packet);
405 static void ssh_ptl_tx_inject_invalid_data(struct ssh_packet *packet)
408 if (!packet->data.ptr || !packet->data.len)
412 if (packet->data.ptr[SSH_MSGOFFSET_FRAME(type)] != SSH_FRAME_TYPE_DATA_SEQ)
418 trace_ssam_ei_tx_corrupt_data(packet);
419 ptl_info(packet->ptl,
420 "packet error injection: simulating invalid transmit data on packet %p\n",
421 packet);
428 memset(packet->data.ptr, 0xb3, packet->data.len);
481 static inline bool ssh_ptl_should_drop_packet(struct ssh_packet *packet)
487 struct ssh_packet *packet,
494 static inline void ssh_ptl_tx_inject_invalid_data(struct ssh_packet *packet)
516 ptl_dbg_cond(p->ptl, "ptl: releasing packet %p\n", p);
521 * ssh_packet_get() - Increment reference count of packet.
522 * @packet: The packet to increment the reference count of.
524 * Increments the reference count of the given packet. See ssh_packet_put()
527 * Return: Returns the packet provided as input.
529 struct ssh_packet *ssh_packet_get(struct ssh_packet *packet)
531 if (packet)
532 kref_get(&packet->refcnt);
533 return packet;
538 * ssh_packet_put() - Decrement reference count of packet.
539 * @packet: The packet to decrement the reference count of.
542 * the packet's &struct ssh_packet_ops, i.e. ``packet->ops->release``, will be
547 void ssh_packet_put(struct ssh_packet *packet)
549 if (packet)
550 kref_put(&packet->refcnt, __ssh_ptl_packet_release);
554 static u8 ssh_packet_get_seq(struct ssh_packet *packet)
556 return packet->data.ptr[SSH_MSGOFFSET_FRAME(seq)];
560 * ssh_packet_init() - Initialize SSH packet.
561 * @packet: The packet to initialize.
562 * @type: Type-flags of the packet.
563 * @priority: Priority of the packet. See SSH_PACKET_PRIORITY() for details.
566 * Initializes the given SSH packet. Sets the transmission buffer pointer to
572 void ssh_packet_init(struct ssh_packet *packet, unsigned long type,
575 kref_init(&packet->refcnt);
577 packet->ptl = NULL;
578 INIT_LIST_HEAD(&packet->queue_node);
579 INIT_LIST_HEAD(&packet->pending_node);
581 packet->state = type & SSH_PACKET_FLAGS_TY_MASK;
582 packet->priority = priority;
583 packet->timestamp = KTIME_MAX;
585 packet->data.ptr = NULL;
586 packet->data.len = 0;
588 packet->ops = ops;
594 * ssh_ctrl_packet_cache_init() - Initialize the control packet cache.
611 * ssh_ctrl_packet_cache_destroy() - Deinitialize the control packet cache.
620 * ssh_ctrl_packet_alloc() - Allocate packet from control packet cache.
621 * @packet: Where the pointer to the newly allocated packet should be stored.
622 * @buffer: The buffer corresponding to this packet.
625 * Allocates a packet and corresponding transport buffer from the control
626 * packet cache. Sets the packet's buffer reference to the allocated buffer.
627 * The packet must be freed via ssh_ctrl_packet_free(), which will also free
629 * separately. Intended to be used with %ssh_ptl_ctrl_packet_ops as packet
634 static int ssh_ctrl_packet_alloc(struct ssh_packet **packet,
637 *packet = kmem_cache_alloc(ssh_ctrl_packet_cache, flags);
638 if (!*packet)
641 buffer->ptr = (u8 *)(*packet + 1);
644 trace_ssam_ctrl_packet_alloc(*packet, buffer->len);
649 * ssh_ctrl_packet_free() - Free packet allocated from control packet cache.
650 * @p: The packet to free.
711 * inserting a control or re-submitted data packet, (determined by
739 static int __ssh_ptl_queue_push(struct ssh_packet *packet)
741 struct ssh_ptl *ptl = packet->ptl;
750 if (test_bit(SSH_PACKET_SF_LOCKED_BIT, &packet->state))
753 /* If this packet has already been queued, do not add it. */
754 if (test_and_set_bit(SSH_PACKET_SF_QUEUED_BIT, &packet->state))
757 head = __ssh_ptl_queue_find_entrypoint(packet);
759 list_add_tail(&ssh_packet_get(packet)->queue_node, head);
763 static int ssh_ptl_queue_push(struct ssh_packet *packet)
767 spin_lock(&packet->ptl->queue.lock);
768 status = __ssh_ptl_queue_push(packet);
769 spin_unlock(&packet->ptl->queue.lock);
774 static void ssh_ptl_queue_remove(struct ssh_packet *packet)
776 struct ssh_ptl *ptl = packet->ptl;
780 if (!test_and_clear_bit(SSH_PACKET_SF_QUEUED_BIT, &packet->state)) {
785 list_del(&packet->queue_node);
788 ssh_packet_put(packet);
806 /* If we are canceling/completing this packet, do not add it. */
813 * On re-submission, the packet has already been added the pending
814 * set. We still need to update the timestamp as the packet timeout is
831 static void ssh_ptl_pending_remove(struct ssh_packet *packet)
833 struct ssh_ptl *ptl = packet->ptl;
837 if (!test_and_clear_bit(SSH_PACKET_SF_PENDING_BIT, &packet->state)) {
842 list_del(&packet->pending_node);
847 ssh_packet_put(packet);
856 ptl_dbg_cond(ptl, "ptl: completing packet %p (status: %d)\n", p, status);
867 * packet to the structures it's going to be removed from.
871 * flag is visible before we actually attempt to remove the packet.
883 static bool ssh_ptl_tx_can_process(struct ssh_packet *packet)
885 struct ssh_ptl *ptl = packet->ptl;
887 if (test_bit(SSH_PACKET_TY_FLUSH_BIT, &packet->state))
891 if (!test_bit(SSH_PACKET_TY_BLOCKING_BIT, &packet->state))
894 /* If we are already waiting for this packet, send it again. */
895 if (test_bit(SSH_PACKET_SF_PENDING_BIT, &packet->state))
904 struct ssh_packet *packet = ERR_PTR(-ENOENT);
910 * If we are canceling or completing this packet, ignore it.
918 * If we cannot process this packet, assume that we can't
919 * process any following packet either and abort.
922 packet = ERR_PTR(-EBUSY);
940 * priority in case the packet is re-submitted (e.g. via
947 packet = p;
952 return packet;
964 ptl_dbg(ptl, "ptl: transmitting sequenced packet %p\n", p);
967 ptl_dbg(ptl, "ptl: transmitting non-sequenced packet %p\n", p);
973 static void ssh_ptl_tx_compl_success(struct ssh_packet *packet)
975 struct ssh_ptl *ptl = packet->ptl;
977 ptl_dbg(ptl, "ptl: successfully transmitted packet %p\n", packet);
980 set_bit(SSH_PACKET_SF_TRANSMITTED_BIT, &packet->state);
983 clear_bit(SSH_PACKET_SF_TRANSMITTING_BIT, &packet->state);
985 /* If the packet is unsequenced, we're done: Lock and complete. */
986 if (!test_bit(SSH_PACKET_TY_SEQUENCED_BIT, &packet->state)) {
987 set_bit(SSH_PACKET_SF_LOCKED_BIT, &packet->state);
988 ssh_ptl_remove_and_complete(packet, 0);
992 * Notify that a packet transmission has finished. In general we're only
993 * waiting for one packet (if any), so wake_up_all should be fine.
998 static void ssh_ptl_tx_compl_error(struct ssh_packet *packet, int status)
1000 /* Transmission failure: Lock the packet and try to complete it. */
1001 set_bit(SSH_PACKET_SF_LOCKED_BIT, &packet->state);
1004 clear_bit(SSH_PACKET_SF_TRANSMITTING_BIT, &packet->state);
1006 ptl_err(packet->ptl, "ptl: transmission error: %d\n", status);
1007 ptl_dbg(packet->ptl, "ptl: failed to transmit packet: %p\n", packet);
1009 ssh_ptl_remove_and_complete(packet, status);
1012 * Notify that a packet transmission has finished. In general we're only
1013 * waiting for one packet (if any), so wake_up_all should be fine.
1015 wake_up_all(&packet->ptl->tx.packet_wq);
1051 static int ssh_ptl_tx_packet(struct ssh_ptl *ptl, struct ssh_packet *packet)
1057 if (unlikely(!packet->data.ptr))
1060 /* Error injection: drop packet to simulate transmission problem. */
1061 if (ssh_ptl_should_drop_packet(packet))
1064 /* Error injection: simulate invalid packet data. */
1065 ssh_ptl_tx_inject_invalid_data(packet);
1067 ptl_dbg(ptl, "tx: sending data (length: %zu)\n", packet->data.len);
1069 packet->data.ptr, packet->data.len, false);
1075 buf = packet->data.ptr + offset;
1076 len = packet->data.len - offset;
1078 status = ssh_ptl_write_buf(ptl, packet, buf, len);
1104 struct ssh_packet *packet;
1107 /* Try to get the next packet. */
1108 packet = ssh_ptl_tx_next(ptl);
1110 /* If no packet can be processed, we are done. */
1111 if (IS_ERR(packet)) {
1116 /* Transfer and complete packet. */
1117 status = ssh_ptl_tx_packet(ptl, packet);
1119 ssh_ptl_tx_compl_error(packet, status);
1121 ssh_ptl_tx_compl_success(packet);
1123 ssh_packet_put(packet);
1130 * ssh_ptl_tx_wakeup_packet() - Wake up packet transmitter thread for new
1131 * packet.
1132 * @ptl: The packet transport layer.
1134 * Wakes up the packet transmitter thread, notifying it that a new packet has
1135 * arrived and is ready for transfer. If the packet transport layer has been
1147 * ssh_ptl_tx_start() - Start packet transmitter thread.
1148 * @ptl: The packet transport layer.
1164 * ssh_ptl_tx_stop() - Stop packet transmitter thread.
1165 * @ptl: The packet transport layer.
1195 struct ssh_packet *packet = ERR_PTR(-ENOENT);
1201 * We generally expect packets to be in order, so first packet
1210 * error completion. The packet will be removed shortly.
1213 packet = ERR_PTR(-EPERM);
1218 * Mark the packet as ACKed and remove it from pending by
1228 packet = p;
1234 return packet;
1237 static void ssh_ptl_wait_until_transmitted(struct ssh_packet *packet)
1239 wait_event(packet->ptl->tx.packet_wq,
1240 test_bit(SSH_PACKET_SF_TRANSMITTED_BIT, &packet->state) ||
1241 test_bit(SSH_PACKET_SF_LOCKED_BIT, &packet->state));
1252 * The packet has not been found in the set of pending
1255 ptl_warn(ptl, "ptl: received ACK for non-pending packet\n");
1258 * The packet is pending, but we are not allowed to take
1266 ptl_dbg(ptl, "ptl: received ACK for packet %p\n", p);
1269 * It is possible that the packet has been transmitted, but the state
1274 * On transmission failure, the packet will be locked after this call.
1280 * The packet will already be locked in case of a transmission error or
1282 * packet.
1286 ptl_err(ptl, "ptl: received ACK before packet had been fully transmitted\n");
1300 * ssh_ptl_submit() - Submit a packet to the transport layer.
1301 * @ptl: The packet transport layer to submit the packet to.
1302 * @p: The packet to submit.
1304 * Submits a new packet to the transport layer, queuing it to be sent. This
1307 * Return: Returns zero on success, %-EINVAL if a packet field is invalid or
1308 * the packet has been canceled prior to submission, %-EALREADY if the packet
1309 * has already been submitted, or %-ESHUTDOWN if the packet transport layer
1319 /* Validate packet fields. */
1352 * __ssh_ptl_resubmit() - Re-submit a packet to the transport layer.
1353 * @packet: The packet to re-submit.
1355 * Re-submits the given packet: Checks if it can be re-submitted and queues it
1356 * if it can, resetting the packet timestamp in the process. Must be called
1359 * Return: Returns %-ECANCELED if the packet has exceeded its number of tries,
1360 * %-EINVAL if the packet has been locked, %-EALREADY if the packet is already
1363 static int __ssh_ptl_resubmit(struct ssh_packet *packet)
1368 lockdep_assert_held(&packet->ptl->pending.lock);
1370 trace_ssam_packet_resubmit(packet);
1372 spin_lock(&packet->ptl->queue.lock);
1374 /* Check if the packet is out of tries. */
1375 try = ssh_packet_priority_get_try(packet->priority);
1377 spin_unlock(&packet->ptl->queue.lock);
1381 status = __ssh_ptl_queue_push(packet);
1384 * An error here indicates that the packet has either already
1388 spin_unlock(&packet->ptl->queue.lock);
1392 packet->timestamp = KTIME_MAX;
1394 spin_unlock(&packet->ptl->queue.lock);
1405 * packets that are out of tires in this function. The packet will be
1406 * eventually canceled and completed by the timeout. Removing the packet
1407 * here could lead to overly eager cancellation if the packet has not
1409 * ssh_ptl_tx_next() removed the packet from the queue and updated the
1419 * Re-submission fails if the packet is out of tries, has been
1434 * ssh_ptl_cancel() - Cancel a packet.
1435 * @p: The packet to cancel.
1437 * Cancels a packet. There are no guarantees on when completion and release
1441 * Note that it is not guaranteed that the packet will actually be canceled if
1442 * the packet is concurrently completed by another process. The only guarantee
1443 * of this function is that the packet will be completed (with success,
1447 * May be called before the packet has been submitted, in which case any later
1448 * packet submission fails.
1458 * Lock packet and commit with memory barrier. If this packet has
1466 * By marking the packet as locked and employing the implicit memory
1468 * the packet cannot be added to the queue any more.
1470 * In case the packet has never been submitted, packet->ptl is NULL. If
1471 * the packet is currently being submitted, packet->ptl may be NULL or
1472 * non-NULL. Due marking the packet as locked above and committing with
1473 * the memory barrier, we have guaranteed that, if packet->ptl is NULL,
1474 * the packet will never be added to the queue. If packet->ptl is
1540 * Re-submission fails if the packet is out of tries, has been
1547 /* Go to next packet if this packet is not out of tries. */
1551 /* No more tries left: Cancel the packet. */
1554 * If someone else has locked the packet already, don't use it
1561 * We have now marked the packet as locked. Thus it cannot be
1563 * We can therefore re-use the pending_node of this packet
1575 /* Cancel and complete the packet. */
1620 * Check if SEQ has been seen recently (i.e. packet was
1627 ptl_dbg(ptl, "ptl: ignoring repeated data packet\n");
1651 struct ssh_packet *packet;
1656 status = ssh_ctrl_packet_alloc(&packet, &buf, GFP_KERNEL);
1658 ptl_err(ptl, "ptl: failed to allocate ACK packet\n");
1662 ssh_packet_init(packet, 0, SSH_PACKET_PRIORITY(ACK, 0),
1667 ssh_packet_set_data(packet, msgb.begin, msgb_bytes_used(&msgb));
1669 ssh_ptl_submit(ptl, packet);
1670 ssh_packet_put(packet);
1675 struct ssh_packet *packet;
1680 status = ssh_ctrl_packet_alloc(&packet, &buf, GFP_KERNEL);
1682 ptl_err(ptl, "ptl: failed to allocate NAK packet\n");
1686 ssh_packet_init(packet, 0, SSH_PACKET_PRIORITY(NAK, 0),
1691 ssh_packet_set_data(packet, msgb.begin, msgb_bytes_used(&msgb));
1693 ssh_ptl_submit(ptl, packet);
1694 ssh_packet_put(packet);
1840 * ssh_ptl_rx_start() - Start packet transport layer receiver thread.
1841 * @ptl: The packet transport layer.
1859 * ssh_ptl_rx_stop() - Stop packet transport layer receiver thread.
1860 * @ptl: The packet transport layer.
1877 * ssh_ptl_rx_rcvbuf() - Push data from lower-layer transport to the packet
1879 * @ptl: The packet transport layer.
1884 * packet layer and notifies the receiver thread. Calls to this function are
1885 * ignored once the packet layer has been shut down.
1888 * success. Returns %-ESHUTDOWN if the packet layer has been shut down.
1905 * ssh_ptl_shutdown() - Shut down the packet transport layer.
1906 * @ptl: The packet transport layer.
1908 * Shuts down the packet transport layer, removing and canceling all queued
1955 * packet as locked an then remove it from the queue (or pending set
1956 * respectively). Marking the packet as locked avoids re-queuing
2014 * ssh_ptl_init() - Initialize packet transport layer.
2015 * @ptl: The packet transport layer to initialize.
2019 * Initializes the given packet transport layer. Transmitter and receiver
2021 * ssh_ptl_rx_start(), after the packet-layer has been initialized and the
2074 * ssh_ptl_destroy() - Deinitialize packet transport layer.
2075 * @ptl: The packet transport layer to deinitialize.
2077 * Deinitializes the given packet transport layer and frees resources