Guangjing Wang

Conference Notes

Mon, 31 Oct 2022 00:00:00 +0000

ACM SenSys’2022
ACM CCS’2022

I find it engaging to document the topics covered in conference sessions as it provides us with a fundamental understanding of the prevailing trends in research areas.

ACM SenSys’2022

RF Sensing
New Applications
Low-power Systems
Lighting and Acoustic Sensing
Machine Learning for Mobile Systems
System for Voice, Speech and Video Processing

ACM CCS’2022

Applied Crypto
- Digital Signatures
- Digital Currencies and Blockchains
- Zero-Knowledge
- Cryptographic Protocols
- Advanced Public-Key Encryption Schemes
- Key Exchange
- Security Multiparty Computation
- Private-Set Intersection
- Symmetric-Key Encryption and Attacks
Software Security
- Fuzzing
- Analysis
- Information Leakage and Access Control
- Hardware-Assisted Defense
- Vulnerability Detection
- Defenses and Virtualization-based Security
Web Security
- Client Side Security
- Server Side Security
- Users Under Attack
Network Security
- Internet Security
- Cloud and IoT Security
- Network Vulnerabilities
Cyber-Physical System
- CPS and IoT Security
- General Attacks and Defense
- Side Channel Attacks
  - Hardware and Microarchitecture
Machine Learning
- Poisoning and Backdooring ML
- Inference Attacks to ML
- ML for Network Security
- Federated Learning Security
- General Attacks to ML
- Adversarial Examples in ML
Privacy and Anonymity
- Secure Query Answering
- Ads and Location Privacy
- Online, Mobile, Multimedia Privacy
- Privacy Attacks in ML
- Privacy Preserving Machine Learning
- Federated Analytics and Learning
- Differential Privacy
- Privacy in Graphs
Usability and Measurement
- Finding Violations
- Attacks
- Security and Privacy Practices
Blockchain and Distributed System
Hardware Security

Writing Notes

Mon, 26 Sep 2022 00:00:00 +0000

Word
Sentence
Paragraph
Email

The goal of good writing is straightforward: to make your reader’s job as easy as possible. The trick is not to copy someone else’s voice, but rather to study what works and what does not in your own writing and that of others to formulate your own guide to effective communication.

In the art of writing science article, the author discusses some of the rules he has picked up for tackling each of these elements.

Word

Jargon is one of the greatest enemies of clear scientific writing.
Do not invent new words or phrases if there is already a commonly used word or phrase that accurately fits the bill.
If at all possible, avoid using abbreviations.
Word choice also impacts the precision of your writing. For example, “these data show” is less precise than “these data indicate” or “these data demonstrate”; although “show” can denote both “indicate” or “demonstrate,” the two words are not interchangeable as the former is synonymous with “suggest” and the latter with “prove”. Using the word “show” thus forces your readers to pause, break their stream of consciousness, and waste effort deciding what you meant.

Sentence

Passive language, although extremely common in scientific writing, is a bore that your readers’ eyes will tend to glide over without taking in.
Be aware of long sentences. While it is good to vary your sentence length across a paragraph, longer sentences are generally harder to parse than their shorter brethren.

Paragraph

The first sentence of each paragraph should tell the reader what you expect them to get out of the paragraph that follows, which makes their job of following it far easier.
The first sentence of a paragraph generally should not refer to an idea in the paragraph above without, at the least, restating that idea. For this reason, I avoid starting paragraphs with phrases such as “However”, or “In contrast”.
A paragraph should discuss only a single idea and thus should have a single, unifying theme running throughout it.
Flow is an equally critical element of a good paragraph structure: every sentence in a paragraph should arise logically from the sentence before it, and transition logically into the sentence that follows.
A scientific paper is composed of specific elements: a title, an abstract, an introduction, methods, results, figures and captions, and, often, a stand‐alone discussion section.
Figure caption should be clear. I believe that the reader should be able to ascertain the entire story just by reading the figure captions. The first sentence of the figure caption should be, in effect, a title for the figure that clearly and concisely tells your readers what you expect them to learn from the figure.

Email

Thank you for the patience and valuable instructions!
With the continuation with previous mail, In the past two weeks, I have gone though multiple data sources including Factiva, Bloomberg, Reuters, analyst reports, industry reports and local news outlets looking for important updates on this topic.
There are multiple interesting aspects that I looked into…
Please find the updated attached excel sheet which contains all the information collected as instructed in your request. Also, In the spreadsheet attached, I have included the keywords and link for each news source, highlighted the cells for the newly added records for the period between 24 Dec- 31 Jan 2022.
Please do let me know your valuable advise on the shared excel sheet. Please don’t hesitate to contact me if you have any questions.

Server Setup (3)

Fri, 15 Oct 2021 00:00:00 +0000

Network
Login
Softwares
MISC

This is a quick guide for using Deep Learning Stack Server in OTML-SEIT labs. The basic configuration of the server is: TensorEX TS4-178672310 - 2x Gold 5218R, 16GBx16 DDR4, 2x 2TB NVMe SSD, 8x 8TB SATA HDD, 8x RTX A6000, Ubuntu 20.04 w/ EMLI Container Deep Learning Stack.

Network

The server is maintained by DECS and OTML-SEIT labs in MSU, so make sure you can access the EGR network in MSU before logging into the server.

Activate your EGR Account before connecting to the server here.
For Windows users, you can find Engineering VPN here. For more information about remote access and VPN setup, please visit DECS.
An alternative option is to use the jump server if you are outside of the MSU network:
```
ssh {netid}@scully.egr.ms.edu
```

Use ssh to log in as below: ssh {netid}@otml-set.egr.ms.edu I miss U :)
/home/{netid}/: Upon logging in you’ll have your usual EGR home directory at /home/{netid}, we do not recommend you do storage in the home folder. Instead, we have two other options.
/localscratch/ and /localscratch2/: Each is individual and non-redundant, but very durable; this way there is roughly 7TB worth of fast storage available for working from. Both of these are where other groups have their users make themselves a folder for their own access (named with their netid for ease of identification)
/storage/: It is slower than the NVMe, but has about 42 TB total usable; it is set up to allow the failure of 2 of the 8 drives without data loss. It is highly recommended that running your data from scratch, and then once results are done and ready to be archived/whatever, you can just copy/move them to /storage.

Softwares

Anaconda

Do not install anaconda in your home folder since it will be large like 3.5G+. Instead, follow conda install to install anaconda. Here is recommended:

/localscratch/{netid}/anaconda3

conda list
conda env list
conda create -n your_env_name python=X.X
conda install -n your_env_name [package]
conda remove -n python36 --all
conda activate your_env_name
source activate your_env_name

Virtualenv

Virtualenv is a tool to create isolated Python environments. Follow virtualenv install to install virtualenv. In most cases, you will not need the sudo permission under virtualenv.

Pytorch and Tensorflow

Follow pytorch install and tensorflow install here to install these two software.

The default GPU for OTML Lab (Prof. Liu) is GPU ‘0’, ‘1’, ‘2’, ‘3’.
The default GPU for SEIT Lab (Prof. Yan) is GPU ‘4’, ‘5’, ‘6’, ‘7’.

Be sure not to use multiple GPUs at the same time if one is enough! If the other lab would like to use their default GPUs, you need to kill the jobs ASAP. Here are some ways to specify the number of GPUs you can use:

export CUDA_VISIBLE_DEVICES=4 # Command example to use GPU '4'
python example.py
****************************************
CUDA_VISIBLE_DEVICES=4 python example.py
****************************************
os.environ["CUDA_VISIBLE_DEVICES"] = "4,5,6,7" # Command example to use GPU '4', must set before 'import torch'
****************************************
config = tf.ConfigProto()
config.gpu_options.allow_growth = True # allocate gpu according to its need
config.gpu_options.per_process_gpu_memory_fraction = 0.4 # specifiy GPU memory fraction
****************************************
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Tmux and Screen

Tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. Tmux may be detached from a screen and continue running in the background, then later reattached. It is recommended to choose Tmux, and you can find a good guide to Tmux HERE.

Here are some example usages:

tmux new -s <session-name>
tmux detach
tmux attach -t <session-name>
tmux kill-session -t <session-name>
tmux switch -t <session-name>

An alternative choice is to use screen. You can start a screen session and then open any number of windows (virtual terminals) inside that session. Processes running in Screen will continue to run when their window is not visible even if you get disconnected. Here are some example usages:

sudo apt install screen
screen -S yourname # create a named session， 
screen -r yourname # reattach to your session
screen -d -r yourname # deattach and reattach to your session
screen -ls # list all sessions

Ctrl-A-Esc to see more content on the screen; Esc to exit the mode;
your_command|less to see more content;

MISC

Here are some other commands you may find helpful.

Git

Here are some git commands in case you may use git on the server:

git status  # check git status
git add -A	# Add all new and changed files to the staging area
git commit -m "[commit message]" # Commit changes
git push -u origin [branch name] # Push changes to the remote repository (and remember the branch)
git diff file # Preview changes before merging
git checkout # Switch to a branch and discard the changes
git reset HEAD file # Discard changes in the staging area

GPU usage

Use the below command for a breakdown of the GPU usage:

nvidia-smi
watch -n 1 nvidia-smi

File Transfer

Here are some recommendations for transferring files:

scp, rz/sz, FileZilla

File System

Ubuntu mount HPCC Reference:

sshfs user-name@hpcc.ms.edu:/mnt/home/user-name/ ./hpcc
sshfs user-name@otml-set.egr.ms.edu:/localscratch/user-name/ ./otml-set

One thing that causes my disk being full is that the ~/.local/share/Trash becomes more than 300 GB.

du: disk usage, '-s' means summary, '-h' means human readable
    du -sh /home/name/* | sort -hr
    du -h --max-depth=1 /home/name/
df: disk free, '-h' means human readable
    df -h

JupyterLab

Install Jupyterlab on the server. You may use conda to install.
Start Jupyterlab with no browser. You may need to activate the environment such as “conda activate dgl”.
```
jupyter lab --no-browser --ip "*" --notebook-dir /localscratch/user-name/
```
Create a Screen: screen -S jupyter
You need to ssh in local terminal: ssh -L 8888:127.0.0.1:8888 user-name@otml-set.egr.ms.edu

I miss U :)

Recycle bin gets problem: dropped file are deleted permanately.

Solution: rd /s /q C:\$Recycle.bin

Server Setup (2)

Sat, 07 Aug 2021 00:00:00 +0000

Installation
Usage

Docker serves as an open platform, enabling developers and system administrators to construct, transport, and execute distributed applications, whether they are on laptops, virtual machines in data centers, or in the cloud.

Installation

Using Docker, developers package their applications and dependencies into a lightweight, portable container, and then publish it to any popular Linux machine, which can be virtualized and has extremely low container performance overhead. Docker images include Running environment and configuration, so Docker can simplify the deployment of multiple application instances. For example, web applications, background applications, database applications, big data applications such as Hadoop clusters, message queues, etc. can be packaged into a mirror for deployment.

Install Docker on Centos: Centos Install Docker on Ubuntu: Ubuntu

uname -r // check kernel version
wget -qO- https://get.docker.com/ | sh　// download installation package
sudo usermod -aG docker runoob 　// commands prompted to enter after installation is complete
sudo service docker start　//start docker service, or using sudo systemctl start docker
docker ps -a　// query container

Usage

“docker run” is only used when running for the first time. Put the image in the container. To start the container again in the future, you only need to use docker start (start the existing image, but you cannot specify the instructions to run when the container starts). , docker run is equivalent to performing two steps, placing the image in the container (docker create), and then starting the container to turn it into a runtime container (docker start).
When using docker run to create a container, the standard operations Docker runs in the background include:
- Check whether the specified image exists locally. If not, download it from the public warehouse.
- Use the image to create and start a container.
- Allocate a file system and mount a read-write layer outside the read-only image layer.
- Bridge a virtual interface from the bridge interface configured on the host host to the container.
- Configure an IP address from the address pool to the container.
- Execute user-specified applications.
- The container is terminated after execution is completed.
You need to ensure that the container is stopped before deleting it. Use the stop command. Please note that the commands for deleting images and containers are different. docker rmi ID, where container (rm) and image (rmi).
Because the ID of the container is a random code, and the name of the container is a seemingly meaningless naming, we can use the command:
```
  docker rename  old_name new_name

  docker [stop] [start]  new_name // we can use new name now
```
List the images on the machine: docker images
Pull down the image or repository from the docker registry server (pull), push an image or repository to the registry (push), corresponding to pull
```
docker pull centos:centos6 // Officical Image
docker pull username/repository<:tag_name> // Personal Image
docker pull seanlook/centos:centos6
```

Use image to create a container and enter interactive mode, login shell is /bin/bash

sudo docker run -it -h pbs –-name pbs -e PBS_START_MOM=1 pbspro/pbspro bash // -i: interactive mode, -t: allocate pseudo terminal, -h: give hostname, --name give a name;
sudo docker  --privileged -p 8817:6817 -p 8818:6818 --dns 8.8.8.8 --dns 8.8.4.4 -h controler --name slurm_control1  -i -t run ubuntu:16.04 /bin/bash
sudo docker run -h controler --name slurm_control1  -i -t ubuntu:16.04 /bin/bash

Mapping host to container port and directory: -p

-p 127.0.0.1:127.0.0.1:11211:11211 // Only bind port 11211 of the localhost interface
-p 11211:11211　// This is the default situation, binding the 11211 port of all network cards (0.0.0.0) of the host to the 11211 port of the container.

Solidify a container into a new image:

docker commit <cotainer> [repo:tag]
docker commit -m "some tools installed" fcbd0a5348ca seanlook/ubuntu:14.10_tutorial

To install software in docker, you need apt update to install the software after entering the terminal. The container installed by Docker’s Ubuntu image does not have the ifconfig command and the ping command to solve the problem:
```
apt-get update
apt install net-tools       // ifconfig 
apt install iputils-ping     // ping
```

Execute the command on the host machine to copy the file to docker

docker cp // The file path to be copied and the container name: must be assigned to the corresponding path in the container;

Server Setup (1)

Sun, 01 Aug 2021 00:00:00 +0000

Gitlab
ownCloud
IP and DNS

In this article, my focus primarily revolves around documenting the procedure for setting up personal servers for ownCloud and Gitlab, aided by the plethora of valuable online resources and blogs available.

Gitlab

GitLab uses Ruby on Rails, an open source version management system, to implement a self-hosted Git project warehouse that can access public or private projects through a web interface. It has similar functionality to Github, with the ability to browse source code and manage defects and comments. Installation reference link: Gitlab. In fact, the installation process is relatively simple.

Software integration is a link in the software development process. The work in this link generally includes the following processes: merge code—>install dependencies—>compile—>test—>release, software The integration work is detailed and tedious and was previously done manually. But now continuous integration is encouraged (continuous integration is a software development practice, that is, team development members integrate their work frequently, usually each member integrates at least once a day, and each integration is through an automated build), therefore, the continuous integration system should be And born. GitLab-CI is a continuous integration system used with GitLab. Generally, each project in GitLab will define a software integration script belonging to this project to automatically complete some software integration work. When the warehouse code of this project changes, for example, someone pushes the code, GitLab will notify GitLab-CI of the change. At this time, GitLab-CI will find the Runners associated with this project and notify these Runners to update the code locally and execute the predefined execution script. Therefore, GitLab-Runner is a tool used to execute software integration scripts.

In fact, in addition to Github and Gitlab, there is also Bitbucket as a code warehouse management.

ownCloud

ownCloud is a tool used to create your own private cloud service, which can fully control the data and can be used within a pure LAN. It supports file preview, version control, link sharing, and can also load third-party storage, API support, and more. Both server and client are supported by all platforms. First, you need to install the LAMP suite; then install ownCoud and configure the database.

Reference: csdnblog digitalocean

IP and DNS

The network outside the school is really complicated. If you have a company IP, you can only apply for a fixed IP from the operator, otherwise it will be dynamic. You can use peanut shells or router domain names to bind dynamic IP, and then set up the DMZ host. Broadband external network IPs are displayed starting with 10, 100, and 172, and there is no public network IP. The reason is that under the current IPV4, public network IP as a non-renewable resource has been stretched, and there are more and more Internet users and Internet devices, so operators have It is impossible to provide sufficient IP address resources for new users, so operators can only use NAT forwarding technology (network address translation) to put some users in a large LAN and share a small number of external network exits, thereby alleviating IP pressure. , just like a community, you and all residents in the same community share a community gate. The solution is of course to file a complaint or spend more money to fix the IP.

Firewall

sudo ufw app list　//registed a few profiles with ufw
sudo ufw status
sudo ufw allow 'Apache Full'
sudo ufw delete allow 'Apache'

Port

One reason is to prevent hackers, and the other reason is that port 80 is blocked by email…

For the Apache web server:

/etc/apache2/ports.conf // change NameVirtualHost *:80 and Listen 80 to our own.
/etc/apache2/sites-available/default // change the VirtualHost :80 to our own e.g., VirtualHost :1000
/etc/init.d/apache2/httpd.conf // add servername localhost

sudo /etc/init.d/apache2 restart 

SSL

To implement https, you need an SSL certificate. Cloud service providers such as Tencent Cloud, Qiniu Cloud, and Alibaba Cloud, but most of them are very expensive. In line with the principle of saving money for the company, I found many ways to make my own certificate, because my 443 port was also blocked. This really made me cry.

Later, I found a [method] for issuing let’s encrypt ssl certificate for non-port 80 that does not support Http verification domain name (https://blog.nbhao.org/2633.html). Finally, it is clear that our problem is to make a self-signed SSL certificate.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
sudo nano /etc/apache2/conf-available/ssl-params.conf
sudo a2enmod ssl
sudo a2enmod headers
sudo a2ensite default-ssl
sudo a2enconf ssl-params
sudo apache2ctl configtest

A new interesting probelm: ERR_SSL_PROTOCOL_ERROR when accessing this server. Soultion reference:

I summarize all procedures to set up ssl and changing its port. In this example, I change ssl port to 18443 from 443.

sudo apt-get install apache2
sudo a2enmod ssl
sudo a2ensite default-ssl
sudo service apache2 restart
ss -lnt

Then, trying to change ssl port:

sudo vi /etc/apache2/ports.conf
<IfModule ssl_module>
        Listen 18443
</IfModule>
<IfModule mod_gnutls.c>
        Listen 18443
</IfModule>

In this setting, I use default-ssl, so I also have to modify this file.

sudo vi /etc/apache2/sites-available/default-ssl.conf
 <IfModule mod_ssl.c>
   <VirtualHost _default_:18443>

Android Notes

Sat, 22 Aug 2020 00:00:00 +0000

Basic Java
Sensors

Some notes about Android Development. I have taken “CSE 476 Mobile Application Development” at MSU, but to avoid leaking course materials, I will just keep them in my mind HOPEFULLY.

Basic Java

JNI: Java Native Interface, which is a feature of Java，making Jave interacts with other programming languages; For example, Java calls C++; C++ calls Java.
NDK: Native Development Kit, which helps to develop C++ dynamic library and automatically package shared libraries and applications into APKs．
Android.mk formulates the configuration information for source code compilation, and Application.mk configures the relevant content of the compilation platform;

Sensors

Location: longitude, latitude, and altitude
Accelerometer: gravity and linear accelerometer
Gyroscope: measuring the rotation speed of the device, mostly required for games like 3D racing
Proximity: check whether there is some obstacle at a very close distance to your device.
Orientation: magnetic field force in 3-axis space, z is related to the magnetic north pole(0 means the top of your phone points to the north pole), x is related to the ground (90 means the phone is standing vertically), y shows roll (0 means the phone is facing to the sky, 180 means the screen is facing the ground).

Celery Notes

Sun, 09 Aug 2020 00:00:00 +0000

Producer-Consumer
Celery with Django
- Add Task
- Periodic Tasks

Celery is a distributed system to process vast amounts of messages while providing operations with the tools requires to maintain such a system. It is a task queue with a focus on real-time processing, while also supporting task scheduling.

Celery architecture consists of three parts: message broker, worker and task result store. Before operating celery in detail, let’s introduce a few concepts.

Producer-Consumer

The producer is the module responsible for generating data; the consumer is the module responsible for processing data. The broker acts as a buffer between the producer and the consumer. The producer puts the data into the buffer, and the consumer takes the data out of the buffer; this buffer plays the role of “decoupling”;

In asynchronous communication, the message will not reach the receiver immediately but will be stored in a container. When certain conditions are met, the message will be sent to the receiver by the container. This container is the message queue. To complete this function, both parties and the container are required to abide by the unified conventions and rules. AMQP (advanced message queuing protocol) is such a protocol, and RabbitMQ is a message-queuing product based on AMQP.

The main elements included in AMQP: are the producer publishes the message; the consumer obtains the message from the message queue; the message queue is used to save the message. The exchanger is the routing component that accepts the message sent by the producer and forwards the message route to the message queue; The virtual host is a collection of switches, message queues, and related objects. A channel is an independent bidirectional data flow channel in a multiplexed connection, providing a physical transmission medium for a session.

publisher--publish-->exchange--routes-->queue--consume-->consumer

Celery with Django

Celery Reference:

celery -A petct worker -l info --pidfile id.txt 
# The --pidfile argument can be set to an absolute path to avoid  having other old workers still running.

Add Task

# tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task
from celery.decorators import task
from celery.utils.log import get_task_logger
from .draw3d import loadArray

logger = get_task_logger(__name__)

@shared_task(name="loadArray")
def loadArray_task(filepath, shape):
    logger.info("load 3d array")
    return loadArray(filepath, shape)



# views.py

from .tasks import loadArray_task

def show(request):
    if request.method == 'GET':
        filePos=request.GET.get("userID", None)
        record=Record.objects.get(userID=filePos)

        global mydict
        if filePos not in mydict:
            loadtask=loadArray_task.delay("./file.mpv", (110,110,187))
            mydict[filePos]=loadtask.get(timeout=1)

        return render(request, 'show.html', {'record': record})

Periodic Tasks

Scheduling a task to run at a specific time, Refer.

# tasks.py
# we run the save_latest_flickr_image() function every fifteen minutes by wrapping the function call in a task. 

from celery.task.schedules import crontab
from celery.decorators import periodic_task
from celery.utils.log import get_task_logger

from photos.utils import save_latest_flickr_image

logger = get_task_logger(__name__)

@periodic_task(
    run_every=(crontab(minute='*/15')),
    name="task_save_latest_flickr_image",
    ignore_result=True
)
def task_save_latest_flickr_image():
    """
    Saves latest image from Flickr
    """
    save_latest_flickr_image()
    logger.info("Saved image from Flickr")

We can also run the job remotely using Supervisor. We need to tell Supervisor about our Celery workers by adding configuration files to the “/etc/supervisor/conf.d/” directory on the remote server. In our case, we need two such configuration files - one for the Celery worker and one for the Celery scheduler.

Redis Notes

Sat, 25 Jul 2020 00:00:00 +0000

Scenarios
Installation
Advanced
Python with Redis

Redis is an in-memory data structure store, used as a database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries, and streams.

Redis supports data persistence. It can save data in memory to disk and load it again for use when restarting. Redis not only supports simple key-value type data, but also provides string, list, set, and zset. , storage of data structures such as hash; Redis supports data backup, that is, data backup in master-slave mode; all operations of Redis are atomic, either executed successfully or not executed at all if failed.

Scenarios

(1) Cache data, for data that needs to be queried frequently and changes infrequently (hot data);

(2) Used as a message queue, equivalent to a message subscription system;

(3) As a counter, count click rate, like rate, etc., because redis is atomic and can avoid concurrency problems;

Why is Redis so fast? Pure memory operation (can be manually persisted to the hard disk), single thread (avoiding frequent context switching operations in multi-threads), the data structure and operation are relatively simple, redis has established its own VM mechanism in the underlying model, because of general system calls For system functions, a certain amount of time will be wasted to move and request, so use a multi-channel I/O multiplexing model and a non-blocking model.

Installation

sudo apt-get update
sudo apt-get install redis-server

redis-server --maxclients 100000 #start redis, set maximum connections

redis-cli # Connect to local redis service 
redis-cli -h host -p port -a password # Execute commands on the remote redis service

String is the most basic type of redis. One key corresponds to one value. It is binary safe and can contain any data, such as pictures or serialized objects. Redis hash is a set of key-value (key=>value) pairs. Hash is particularly suitable for storing objects. For example, it can be used to store user IDs, avatar points, etc. in forum systems. If you want to modify the information, you only need to retrieve it through the key. Value is deserialized to modify the value of one of the items, and then serialized and stored in Redis. Redis list is sorted in insertion order. You can add an element to the head (left) or tail (right) of the list. Set is an unordered collection of string type, implemented through a hash table. Addition, deletion, and search complexity are all O(1).

SET string_name "runoob"
GET string_name

HMSET myhash key_field1 "Hello" key_field2 "World"
HGET myhash key_field1

lpush list_runoob redis
lrange list_runoob 0 10

sadd set_runoob redis
smembers set_runoob

zadd key score member 

Redis HyperLogLog

Data set {1, 3, 5, 7, 5, 7, 8}, then the cardinality set of this data set is {1, 3, 5,7, 8}, and the cardinality (non-repeating elements) is 5. Cardinality estimation is to quickly calculate the cardinality within the acceptable error range. Redis HyperLogLog is an algorithm used for cardinality statistics. When the number or volume of input elements is very large, the space required to calculate the cardinality is always fixed and very small.

Advanced

Redis publish and subscribe (pub/sub) is a message communication model: the sender (pub) sends messages and the subscriber (sub) receives messages:

SUBSCRIBE redisChat
PUBLISH redisChat "Learn redis by runoob.com"

Redis transactions can execute multiple commands at one time. Batch operations are put into the queue cache before sending the EXEC command. After receiving the EXEC command, transaction execution is entered. If any command in the transaction fails to execute, the remaining commands are still executed. During transaction execution, command requests submitted by other clients will not be inserted into the transaction execution command sequence.

MULTI # start transaction
SET book-name "Mastering C++ in 21 days"　# command to queue
SADD tag "C++" "Programming" "Mastering Series"　# command to queue
EXEC　# execute transaction

The Redis SAVE command is used to create a backup of the current database; BGSAVE, this command is executed in the background; if you need to restore data, just move the backup file (dump.rdb) to the redis installation directory and start the service.

CONFIG GET dir # get file directory

Deletion

The default is to detect once every 100ms. If an expired key is encountered, it will be deleted. Random detection is used here. In order to prevent the fish from slipping through the net, when we read and write an expired key, the lazy deletion strategy of redis will be triggered. Directly Expired keys will be deleted.

Breakdown

Caching is a layer of protection added to relieve the pressure on the database. When the data we need cannot be queried from the cache, we must query the database. If hackers use it to frequently access data that does not exist in the cache, the cache will lose its meaning. , causing database connection exception;

Solution:

Set up scheduled tasks in the background and actively update cached data;
Hierarchical cache, for example, setting up two cache protection layers. The first-level cache has a short expiration time, and the second-level cache has a long expiration time. When a request comes, priority is given to searching from the first-level cache. If the corresponding data is not found in the first-level cache, Then the thread is locked, and this thread gets the data from the database, updates the level 1 and level 2 caches, and other threads get it directly from the level 2 cache.
Provide an interception mechanism that internally maintains a series of legal key values. When the requested key is illegal, it will be returned directly.

Cons

Since Redis is an in-memory database, the amount of data stored in a single machine is limited, and unnecessary data needs to be deleted in time. After modifying the redis data, the data persisted to the hard disk needs to be re-added to the memory, which takes a long time. redis cannot run normally.

Python with Redis

import redis
r = redis.Redis(host='192.168.0.110', port=6379,db=0)

pool = redis.ConnectionPool(host='192.168.0.110', port=6379)
r = redis.Redis(connection_pool=pool)

pipe = r.pipeline(transaction=True) # use pipline to specify multiple commands in one request

r.set('name', 'zhangsan')
r.mset(name1='zhangsan', name2='lisi')

r.hset("dic_name","a1","aa")　# name, key, value
r.hget("dic_name","a1")

r.lpush(name,values)
r.rpush(name,values)
r.lpop(name)

r.sadd(name,values)

r.zadd("zset_name", "a1", 6, "a2", 2,"a3",5)

class RedisHelper(object):
    def __init__(self):
        self.__conn = redis.Redis(host='192.168.0.110',port=6379)
        self.channel = 'monitor'

    def publish(self,msg):
        self.__conn.publish(self.channel,msg)
        return True

    def subscribe(self):
        pub = self.__conn.pubsub()
        pub.subscribe(self.channel)
        pub.parse_response()
        return pub

from RedisHelper import RedisHelper

obj = RedisHelper()
obj.publish('hello')

obj = RedisHelper()
redis_sub = obj.subscribe()

while True:
    msg= redis_sub.parse_response()
    print (msg)

Django Notes

Sat, 18 Jul 2020 00:00:00 +0000

Django Introduction
Reset Django migration
Django admin
Django View
UI in Chinese
jQuery
Others

Django, a Python web framework known for its emphasis on swift development, has prompted me to jot down some notes during my Django usage. Additionally, I have taken notes on JQuery.

Django Introduction

Django is an MVC model: (i) M means Model, which defines the database-related content and is placed in the model.py file. (ii) V means View, which defines the front-end related to the static web pages such as HTML; (iii) C means Controller, which defines the processing logic in the view.py file. A Django project can have multiple apps, which can be regarded as sub-systems/sub-modules in a large-scale project. These apps can be independent but could be related.

Reset Django migration

Delete all migrations, and then delete the database:

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete

Re-generate migrations

python manage.py makemigrations
python manage.py migrate

Django admin

class showAdmin(admin.ModelAdmin):
    list_display = ('userID', 'name', 'taskID', 'status', 'createTime', 'is_sure', 'show_firm_url',)
    search_fields = ('name',)
    list_display_links = ('userID', 'status')

    def show_firm_url(self, obj):
        return format_html("<a href='{url}'>Download</a>", url="/mtasks/download/")
    show_firm_url.short_description = "Results"

    def get_readonly_fields(self, request, obj=None):
        if hasattr(obj, 'is_sure'):
            if obj.is_sure:
                self.readonly_fields = ('is_sure', 'userID', 'name', 'taskID', 'status')
        else:
            self.readonly_fields = ('userID', 'taskID', 'status')
        return self.readonly_fields
 
    def change_view(self, request, object_id, form_url='', extra_context=None):
        change_obj = self.model.objects.filter(pk=object_id)
        self.get_readonly_fields(request, obj=change_obj)
        return super(showAdmin, self).change_view(request, object_id, form_url, extra_context=extra_context)

    def save_model(self, request, obj, form, change):
    def delete_model(self, request, obj):

Django View

def get_pic(request):
    if request.method == 'POST':
        fileID=request.POST.get("fileID")
        foldPath='results/'+str(fileID)+'/'
        if not os.path.exists(settings.PICTURE_URL+fileID+"snap_x.gif"):
            drawGIF(foldPath+'voxel_0_subset_0.mpv', 'x', fileID)
            drawGIF(foldPath+'voxel_0_subset_0.mpv', 'y', fileID)
            drawGIF(foldPath+'voxel_0_subset_0.mpv', 'z', fileID)
        pic1=fileID+"snap_x.gif";
        pic2=fileID+"snap_y.gif";
        pic3=fileID+"snap_z.gif";
        data=[pic1, pic2, pic3]
        return HttpResponse(json.dumps(data), content_type='application/json')
// In url.py 
urlpatterns += static('mtasks/download/', document_root=settings.PICTURE_ROOT)

UI in Chinese

SITE_HEADER = os.environ.get('SITE_HEADER', 'Management System') //setting.py
admin.AdminSite.site_header ='Management System'　//admin.py
admin.AdminSite.site_title = 'Management Platform' //admin.py
// models.py
class Meta:
        verbose_name="Task"  //verbose_name can be used for naming table attributes
        verbose_name_plural="Task"

jQuery

Js is a front-end script, and we can use ajax (Asynchronous Javascript and XML) to make the back-end server call python script, and then return results to the front end.jQuery is one of the frameworks of javascript, and jQuery uses ‘$’ as the abbreviation of jQuery.

$(selector).action() //$ defines jQuery;
$(“#test”).hide() //hide all elements with id=”test”;

// To prevent the running of the jQuery code before the full loading of the document.
$(document).ready(function(){
});

load() means loading data from the server and then returning the data to the selected element; $(selector).load(URL, data, callback);
- The required URL parameter specifies the URL to be loaded.
- The optional data parameter specifies the set of the query string (in key-value pairs) sent with the request;
- The optional callback is the name of the function to be executed after the load() method completes;
post() method request to load data from the server with HTTP POST $.post(URL, data, callback); $.post(URL, data, success(data, testStatus, jqXHR), dataType)
- The URL is the address that the request is sent;
- The data can be sent with the request, optional;
- The success() is the callback function after the data is loaded.
- dataType() is the expected type of data from the server.
$.get(url, callback); GET-requesting data from the specified location; POST-sending the data to the specified resources and then requesting data.
$(selector).each(function(index, element)) Specifing the function to run for each matched element.

Others

The decorator @route() can bind a function to the corresponding URL; instead of accessing the index page, the user can directly access the desired page.
Passing parameters with “?” http://127.0.0.1:8000/plist/?p1=china&p2=2012
Show pictures with Django Reference

Monitoring the event:

event.target //return the DOM element
event.target.id
event.target.nodeName
event.target.innerHTML //return the content of the DOM

Example

<script>
  $(document).ready(function(){
    var ide=document.getElementsByName("myUser");
    for (var i=0; i<ide.length; i++){

        $("#"+ide[i].innerHTML).click(function(event){
        url = "/get_pic/";
        alert("Drawing")
        $.post(url, {"fileID":event.target.id}, success, "json");

        function success(data){
          $('#result_x').html('')
          $('#result_y').html('')
          $('#result_z').html('')

          $('#result_x').append('<img src='+data[0] +' alt="Lights" style="width:100%">');
          $('#result_y').append('<img src='+data[1] +' alt="Lights" style="width:100%">');
          $('#result_z').append('<img src='+data[2] +' alt="Lights" style="width:100%">');
        }
      });
    } 
  });
</script>

TensorFlow Notes

Sun, 20 Oct 2019 00:00:00 +0000

Main Changes
Using GPU
Training step
Custom Layer
Custom Model
Basics for 1.0
Optimization
Save Model
Load Model

TensorFlow is an open source software library for high performance numerical computation, and it is developed into TF2.0. Here I add some new notes about tensorflow new features, for more details, you can refer to the official website.

Main Changes

In TF2.0, the eager execution dynamic graph is used by default in TF2.0, and the calculation will be executed directly. If we need to build a static computation graph, we only need to add the @tf.function decorator to the python function. An important change in TF2 is to remove tf.Session(). The more complex the computation graph is, the more execution efficiency and acceleration effect of using static graphs.
The tool that converts python code into graph representation code is called autoGraph. If a function is decorated with @tf.function, then autoGraph will be called automatically, thus converting the python function into a static executable graph representation.
When the function decorated with @tf.function is called for the first time, the function is executed and tracked, eager will be disabled in this function, so each tf.API will only define one to generate tf.Tensor output node.
When using tf.function, you need to check if the function is feasible in a static graph when converting a dynamic graph into a static graph. In a dynamic graph, tf.variable is an ordinary python variable. It will be destroyed beyond its scope. In the static graph, tf.valuable is a persistent node in the calculation graph, which is not affected by the scope of python. We can use tf.variable as a parameter of the function Pass in, call tf.variable as a class attribute.

(1) Every tf.Session.run call should be replaced by a Python function.
    The feed_dict and tf.placeholders become function arguments.
    The fetches become the function's return value.
(2) Use tf.Variable instead of tf.get_variable.
(3) Prefer tf.keras.Model.fit over building your own training loops.
(4) Use tf.data datasets for data input.

import tensorflow_datasets as tfds
datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)
mnist_train, mnist_test = datasets['train'], datasets['test']
def scale(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255
    return image, label
train_data = mnist_train.map(scale).shuffle(BUFFER_SIZE).batch(BATCH_SIZE).take(5)
test_data = mnist_test.map(scale).batch(BATCH_SIZE).take(5)

Using GPU

We can specify the running device by tf.device(). For example, “/cpu:0” is the cpu of your machine, “/device:GPU:0”:The first GPU of your machine, “/device:GPU:1”:The second GPU of your machine.

# specify in code, or you can set when you run the python file
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "2"

# allocate gpu according to its need
config = tf.ConfigProto()
config.gpu_options.allow_growth = True

# specifiy GPU memory fraction
config.gpu_options.per_process_gpu_memory_fraction = 0.4

# TF2.0
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    tf.config.experimental.set_visible_devices(gpus[0], 'GPU')

Training step

If you need more flexibility and control, you can have it by implementing your own training loop. There are three steps:

Iterate over a Python generator or tf.data.Dataset to get batches of examples.
Use tf.GradientTape to collect gradients.
Use a tf.keras.optimizer to apply weight updates to the model’s variables.

Calling minimize() takes care of both computing the gradients and applying them to the variables. If you want to process the gradients before applying them you can instead use the optimizer in three steps:

Compute the gradients with tape.gradient().
Process the gradients as you wish.
Apply the processed gradients with apply_gradients().

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, 3, activation='relu',
                           kernel_regularizer=tf.keras.regularizers.l2(0.02),
                           input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dropout(0.1),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(10, activation='softmax')
])

optimizer = tf.keras.optimizers.Adam(0.001)
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()

# Create the metrics
loss_metric = tf.keras.metrics.Mean(name='train_loss')
accuracy_metric = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

@tf.function
def train_step(inputs, labels):
    with tf.GradientTape() as tape:
      predictions = model(inputs, training=True)
      regularization_loss = tf.math.add_n(model.losses)
      pred_loss = loss_fn(labels, predictions)
      total_loss = pred_loss + regularization_loss

    gradients = tape.gradient(total_loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    # Update the metrics
    loss_metric.update_state(total_loss)
    accuracy_metric.update_state(labels, predictions)


for epoch in range(NUM_EPOCHS):
    # Reset the metrics
    loss_metric.reset_states()
    accuracy_metric.reset_states()

    for inputs, labels in train_data:
      train_step(inputs, labels)
    # Get the metric results
    mean_loss = loss_metric.result()
    mean_accuracy = accuracy_metric.result()

    print('Epoch: ', epoch)
    print('  loss:     {:.3f}'.format(mean_loss))
    print('  accuracy: {:.3f}'.format(mean_accuracy))

Custom Layer

class MyDenseLayer(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
      super(MyDenseLayer, self).__init__()
      self.num_outputs = num_outputs
    
    def build(self, input_shape):
      self.kernel = self.add_variable("kernel", shape=[int(input_shape[-1]), self.num_outputs])
    
    def call(self, input):
      return tf.matmul(input, self.kernel)
  
layer = MyDenseLayer(10)
print(layer(tf.zeros([10, 5])))
print(layer.trainable_variables)

Custom Model

class MNISTModel(Model):
    def __init__(self):
        super(MNISTModel, self).__init__()
        self.conv1 = Conv2D(32, 3, activation='relu')
        self.flatten = Flatten()
        self.d1 = Dense(128, activation='relu')
        self.d2 = Dense(10, activation='softmax')

    def call(self, x):
        x = self.conv1(x)
        x = self.flatten(x)
        x = self.d1(x)
        return self.d2(x)

Basics for 1.0

It is still possible to run 1.X code, unmodified (except for contrib), in TensorFlow 2.0:

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

The tf.layers module is used to contain layer-functions that relied on tf.variable_scope to define and reuse variables.
If you were using regularizers of initializers from tf.contrib, these have more argument changes than others.
# replace tf.contrib.layers.
tf.keras.layers.Conv2D(32, 3, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.04), input_shape=(28, 28, 1))

import tensorflow as tf

x=tf.placeholder(tf.float32, shape=...)
y=tf.variable(0, name='x', dtype=tf.float32)
z=tf.constant(2.0, shape=..., dtype=...)
w=tf.get_variable('w', [1,2,3], initializer=tf.contrib.layers.xavier_initializer(seed=0))

tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=True, data_format='NHWC', dilations=[1, 1, 1, 1], name=None)

tf.nn.max_pool(value, ksize=[1,f,f,1], strides=[1,s,s,1], padding, data_format='NHWC', name=None) #ksize: kernel size, a window of size(f,f), stride of size(s,s)

tf.contrib.layers.flatten(inputs, outputs_collections=None, scope=None) # input is a tensor of size[batch_size,...], return a flattened tensor with shape [batch_size, k].

tf.contrib.layers.fully_connected(
    inputs,
    num_outputs,
    activation_fn=tf.nn.relu,
    normalizer_fn=None,
    normalizer_params=None,
    weights_initializer=initializers.xavier_initializer(),
    weights_regularizer=None,
    biases_initializer=tf.zeros_initializer(),
    biases_regularizer=None,
    reuse=None,
    variables_collections=None,
    outputs_collections=None,
    trainable=True,
    scope=None
) # fully_connected creates a variable called weights, representing a fully connected weight matrix, which is multiplied by the inputs to produce a Tensor of hidden units, returns the tensor variable representing the result of the series of operations. The input is a tensor of at least rank 2 and static value for the last dimension; i.e. [batch_size, depth], [None, None, None, channels]


tf.nn.softmax_cross_entropy_with_logits(_sentinel=None, labels=None, logits=None, dim=-1, name=None)

tf.math.reduce_mean(input_tensor, axis=None, keepdims=None, name=None, reduction_indices=None, keep_dims=None) #Reduces input_tensor along the dimensions given in axis, if keepdims is true, the reduced dimensions are retained with length 1.

tf.reshape(x, [-1, 28, 28, 1])
tf.split(x,n,0) #ｘ is the Tensor to split, n is the number of splits, 0 is the dimension along which to split.

tf.nn.depthwise_conv2d(input, filter, strides, padding, rate=None, name=None, data_format=None) # Must have strides = [1, stride, stride, 1], Given a 4D input tensor and a filter tensor of shape [filter_height, filter_width, in_channels, channel_multiplier], containing in_channels convolutional filters of depth 1, depthwise_conv2d applies a different filter to each input channel (expanding from 1 channel to channel_multiplier channels for each), then concatenates the results together. The output has in_channels * channel_multiplier channels.

Optimization

Gradient Descent: taking gradient step with respect to all m examples on each step
Stochastic GD: using only 1 training examples before updating the gradients
Minibatch GD: using a small part of examples
Momentum: taking into account the past gradients to smooth out the update, and storing the direction of the previous gadients.This will be the exponentially weighted average of gradient on previous steps.
Adam: combing ideas from RMSProp and Momentum.

Save Model

checkpoints is a binary file，which mapping variables into corresponding tensors．

saver=tf.train.Saver(...variables...)
sess = tf.Session()
for step in xrange(1000000):
    sess.run(..training_op..)
    if step % 1000 == 0:
        # Append the step number to the checkpoint name:
        saver.save(sess, 'my-model', global_step=step)   #  ==> filename: 'my-model-[step]'

tf.app.flags get parameters from the coommand line:

flags = tf.app.flags
FLAGS = flags.FLAGS
flags.DEFINE_float('learning_rate', 0.01, 'Initial learning rate.')

From stackoverflow, note that this module is currently packaged as a convenience for writing demo apps, and is not technically part of the public API, so it may change in future. We recommend that you implement your own flag parsing using argparse or whatever library you prefer.

Load Model

# freeze model

pickle.dump(predictions, open("predictions.p", "wb"))
pickle.dump(history, open("history.p", "wb"))
tf.train.write_graph(sess.graph_def, '.', 'har.pbtxt')
saver.save(sess, save_path="./har.ckpt")
sess.close()

from tensorflow.python.tools import freeze_graph

MODEL_NAME = 'har'

input_graph_path = MODEL_NAME + '.pbtxt'
checkpoint_path = MODEL_NAME + '.ckpt'
restore_op_name = "save/restore_all"
filename_tensor_name = "save/Const:0"
output_frozen_graph_name = 'frozen_' + MODEL_NAME + '.pb'

freeze_graph.freeze_graph(input_graph_path, input_saver="",
                          input_binary=False, input_checkpoint=checkpoint_path,
                          output_node_names="y_", restore_op_name="save/restore_all",
                          filename_tensor_name="save/Const:0",
                          output_graph=output_frozen_graph_name, clear_devices=True, initializer_nodes="")

#Load Model

with tf.Graph().as_default():
    output_graph_def = tf.GraphDef()

    with open(pb_file_path, "rb") as f:
        output_graph_def.ParseFromString(f.read())
        _ = tf.import_graph_def(output_graph_def, name="")

    with tf.Session() as sess:
        init = tf.global_variables_initializer()
        sess.run(init)
        input_x = tf.get_default_graph().get_tensor_by_name("input:0")
        out_softmax = sess.graph.get_tensor_by_name("prob:0")
        out_result = sess.run(out_softmax, feed_dict={input_x: reshaped_segments})

Guangjing Wang

Conference Notes

ACM SenSys’2022

ACM CCS’2022

Writing Notes

Word

Sentence

Paragraph

Email

Server Setup (3)

Network

Login

Softwares

Anaconda

Virtualenv

Pytorch and Tensorflow

Tmux and Screen

MISC

Git

GPU usage

File Transfer

File System

JupyterLab

Windows Related

Server Setup (2)

Installation

Usage

Server Setup (1)

Gitlab

ownCloud

IP and DNS

Firewall

Port

SSL

Android Notes

Basic Java

Sensors

Celery Notes

Producer-Consumer

Celery with Django

Add Task

Periodic Tasks

Redis Notes

Scenarios

Installation

Advanced

Deletion

Breakdown

Cons

Python with Redis

Django Notes

Django Introduction

Reset Django migration

Django admin

Django View

UI in Chinese

jQuery

Others

TensorFlow Notes

Main Changes

Using GPU

Training step

Custom Layer

Custom Model

Basics for 1.0

Optimization

Save Model

Load Model