Google Cloud  

Pub/Sub

Rich Lee

2021/8

Integration Architect

  • Enterprise Integration 
  • Microservices
  • Event-Driven Architecture
  • Cloud Native App Development

rich.lee@cathayholdings.com

@rich04230

RICH0423

Agenda

  • Pub/Sub Pattern

  • Google Cloud Pub/Sub

  • Using Cloud Pub/Sub on Java

  • Advanced Concepts 

What is Pub/Sub?

Pub/Sub Pattern

Publish–subscribe is a async messaging pattern where senders of messages

  • Publisher sends the messages
  • Subscriber receives the messages
  • A Channel is a category/feed name to which messages are stored and published

 

one to one

one to many

Pub/Sub Pattern

  • AWS SQS & SNS

  • AWS Kinesis

  • Google Cloud Pub/Sub

  • Azure Service Bus

  • ActiveMQ

  • Redis Pub/Sub

  • Kafka

Pub-Sub / Message Brokers

Cloud Pub/Sub

Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages independent applications.

Cloud Pub/Sub

Google Cloud Services

Cloud Pub/Sub

Cloud Pub/Sub Core Concepts

  • Topic: A named resource to which messages are sent by publishers.
  • Subscription: a named resource that represents an interest in receiving messages on a particular topic.
  • Acknowledgement (ack): A signal sent by a subscriber to Pub/Sub after it has received a message successfully. Acked messages are removed from the subscription's message queue.

 

Cloud Pub/Sub

publisher

subscriber

ACK

Cloud Pub/Sub

Publisher-subscriber relationships

Cloud Pub/Sub

Cloud Pub/Sub Quickstarts

Cloud Pub/Sub Topic:

  • create a topic                                                                                                                   

 

  • list your topics                                                                                                                     
  • create a subscription                                                                                                     

 

gcloud pubsub topics create topic1
gcloud pubsub topics list
gcloud  pubsub subscriptions create sub1 \ 
--topic topic1

Cloud Pub/Sub Subscription:

  • list the subscriptions to topic1                                                                                                               
  • publish the message  to the topic                                                                                                         

 

  • full messages from the subscription                                                                                                     

 

gcloud pubsub topics list-subscriptions topic1
gcloud pubsub topics publish topic1 \
--message "Hello pub/sub"
gcloud pubsub subscriptions pull sub1 --auto-ack

Cloud Pub/Sub Quickstarts

 Using Cloud Pub/Sub

in Java Appplication 

 Using Cloud Pub/Sub in Java Appplication 

Google Pub/Sub Client Lib

Install the client lib

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>20.9.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
  </dependency>

</dependencies>

Google PubSub Client Lib - Publishing messages               

 

public class PublishService {
    private static final String PROJECT_ID = "sample1";
    private static final String TOPIC_NAME = "topic1";

    public void publishFeedback(Feedback feedback) throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        String feedbackMessage = mapper.writeValueAsString(feedback);

        TopicName topicName = TopicName.create(PROJECT_ID, TOPIC_NAME);
        Publisher publisher = null;
        ApiFuture<String> messageIdFuture = null;
        
        try {
            publisher = Publisher.defaultBuilder(topicName).build();
            
            ByteString data = ByteString.copyFromUtf8(feedbackMessage);
            PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).build();
            messageIdFuture = publisher.publish(pubsubMessage);
        } finally {
            String messageId = messageIdFuture.get();

            if (publisher != null) {
                // When finished with the publisher, shutdown to free up resources.
                publisher.shutdown();
            }
        }
    }
}

Google PubSub Client Lib- Receiving messages               

 

 

 

public static void subscribeAsyncExample(String projectId, String subscriptionId) {
    ProjectSubscriptionName subscriptionName =
        ProjectSubscriptionName.of(projectId, subscriptionId);

    // Instantiate an asynchronous message receiver.
    MessageReceiver receiver =
        (PubsubMessage message, AckReplyConsumer consumer) -> {
          // Handle incoming message, then ack the received message.
          System.out.println("Id: " + message.getMessageId());
          System.out.println("Data: " + message.getData().toStringUtf8());
          consumer.ack();
        };

    Subscriber subscriber = null;
    try {
      subscriber = Subscriber.newBuilder(subscriptionName, receiver).build();
      // Start the subscriber.
      subscriber.startAsync().awaitRunning();
      System.out.printf("Listening for messages on %s:\n", subscriptionName.toString());
      // Allow the subscriber to run for 30s unless an unrecoverable error occurs.
      subscriber.awaitTerminated(30, TimeUnit.SECONDS);
    } catch (TimeoutException timeoutException) {
      // Shut down the subscriber after 30s. Stop receiving messages.
      subscriber.stopAsync();
    }
}

Demo project

 

Install the PubSub Starter

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
</dependency>

Spring Boot PubSub Starter

Spring Boot PubSub Starter

Publishing messages   

public void publishMessage() {
        this.pubSubTemplate.publish("topic", "your message payload", 
                ImmutableMap.of("key1", "val1"));
}

Receiving messages

  • Async - subscribe()                     

         


  • Sync - pull()                                


Subscriber subscriber = this.pubSubTemplate.subscribe(subscriptionName, (message) -> {
    log.info("Message received from {} subscription: {}",
             subscriptionName, message.getPubsubMessage().getData().toStringUtf8());
    message.ack();
});
Collection<AcknowledgeablePubsubMessage> messages = this.pubSubTemplate.pull(
        subscriptionName, 10, true);
        
ListenableFuture<Void> ackFuture = this.pubSubTemplate.ack(messages);
ackFuture.get();
messages.stream().
        map(AcknowledgeablePubsubMessage::getPubsubMessage).
        forEach(m -> log.info("MessageId: {}, data: {}", m.getMessageId(),
                m.getData().toStringUtf8()));

Spring Boot PubSub Starter

 Advanced Concepts 

Advanced Concepts

  • Dead Letter Topic
  • Ordering Messages
  • Delivery Semantics

Dead Letter Topic

  • If the Pub/Sub service attempts to deliver a message but the subscriber can't acknowledge it, Pub/Sub can forward the undeliverable message to a dead-letter topic.
    • Create or update a subscription and set the dead-letter topic.
    • maximum number of delivery attempts(5-100 delivery attempts)

Dead Letter Topic

Ordering Messages

  • Messages in the queue is in no order
  • If messages have the same ordering key and you publish the messages to the same region, subscribers can receive the messages in order.

Delivery Semantics

  • Pub/Sub delivers each published message at least once for every subscription