pax_global_header00006660000000000000000000000064141213433560014514gustar00rootroot0000000000000052 comment=7e6595697216259e67e1dbf0e1ab09b8965065d5 growingio-tools-python-master/000077500000000000000000000000001412134335600170325ustar00rootroot00000000000000growingio-tools-python-master/importer/000077500000000000000000000000001412134335600206735ustar00rootroot00000000000000growingio-tools-python-master/importer/.gitignore000066400000000000000000000000731412134335600226630ustar00rootroot00000000000000# default __pycache__/ # intellij ../.idea # config logs/growingio-tools-python-master/importer/README.md000066400000000000000000000236211412134335600221560ustar00rootroot00000000000000# GrowingIO Importer GrowingIO Importer是GrowingIO CDP平台元数据创建和数据导入工具。 ## 入门 有关GrowingIO Importer请访问[GrowingIO官方文档](https://docs.growingio.com/op/developer-manual/toolbox/ )获取帮助。 ## 环境依赖 | |1.0| |----|----| |Python|3.6| ## 安装 下载工具包GrowingIO Importer,在importer目录下执行脚本sh init.sh完成初始化 ## 元数据导入 目前支持如下： * 创建事件 * 创建事件属性 * 创建用户属性 * 绑定事件与事件属性 * 导出元数据 * 导入元数据 ### 使用说明 #### 创建事件 python3 meta_importer -m create_event \ --key <事件名> \ [-n <事件显示名>] \ [-d <事件描述>] \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-k、--key|必选参数，事件名。仅允许大小写英文、数字、以及下划线，并且不能以数字开头，限长30字符| |-n、--name|可选参数，事件显示名。默认同事件名，限长30字符| |-d、--desc|可选参数，事件描述，默认为空。若描述中有空格则需要加双引号| #### 创建事件属性 python3 meta_importer.py -m create_event_variables \ -k <事件名> \ -t <事件属性数据类型> \ [-n <事件显示名>] \ [-d <事件属性描述>] \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-k、--key|必选参数，事件名。仅允许大小写英文、数字、以及下划线，并且不能以数字开头，限长30字符| |-t、--type|必选参数，事件属性数据类型。可选值：string/int/double |-n、--name|可选参数，事件显示名。默认同事件名，限长30字符| |-d、--desc|可选参数，事件描述，默认为空。若描述中有空格则需要加双引号| #### 创建用户属性 python3 meta_importer.py -m create_user_variables \ -k <标识符> \ -t <用户属性数据类型> \ [-n <用户显示名>] \ [-d <用户属性描述>] \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-k、--key|必选参数，标识符。仅允许大小写英文、数字、以及下划线，并且不能以数字开头，限长30字符| |-t、--type|必选参数，可选参数：string/int/date |-n、--name|可选参数，用户显示名。默认同标识符，限长30字符| |-d、--desc|可选参数，用户属性描述，默认为空。若描述中有空格则需要加双引号| #### 绑定事件与事件属性 python3 meta_importer.py -m bind_event_variables \ -k <事件名> \ -a <绑定事件属性名> \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-k、--key|必选参数，事件名。若事件不存在则创建，否则更新事件| |-a、--attr|必选参数，绑定事件属性名。多个属性名使用英文逗号分隔| #### 导出元数据 python3 meta_importer.py -m export_meta \ -f <文件名> \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-f、--file|必选参数，导出文件名| #### 导入元数据 python3 meta_importer.py -m import_meta \ -f <文件名> \ |参数|参数说明| |----|----| |-m|必选参数，项目名。| |-f、--file|必选参数，导入文件名| ## 数据导入 目前支持如下： * 用户属性数据导入 * 用户行为数据导入 ### 使用说明 #### 用户属性数据导入 python3 data_importer.py -m user_variables \ -p 文件路径 \ -ds 数据源ID \ -f [CSV|TSV|Json] \ -d [True|False] \ -qf " \ -sep , \ -skh \ -attr userId,... |参数|参数说明| |----|----| |-m|必填参数. 用户属性数据导入-user_variables| |-p|必填参数. 需要导入的数据所在的路径| |-ds|必填参数. 数据源ID| |-f|可选参数. 导入数据格式,目前支持JSON,CSV,TSV三种格式.默认值:JSON| |-qf|可选参数. CSV,TSV格式文本限定符.默认值:"| |-sep|可选参数. CSV,TSV格式文本分割符.默认值:,| |-skh|可选参数. CSV,TSV格式设置则自动跳过首行,此参数不需要设置值.| |-attr|可选参数. CSV,TSV格式导入文件的各列按顺序映射到属性名，逗号分隔.userId必须指定| #### 用户行为数据导入 python3 data_importer.py -m events \ -p 文件路径 \ -ds 数据源ID \ -f [CSV|TSV|Json] \ -d [True|False] \ -qf " \ -sep , \ -skh \ -attr userId,... \ -s YYYY-MM-DD \ -e YYYY-MM-DD |参数|参数说明| |----|----| |-m|必填参数. 用户行为数据导入-events| |-p|必填参数. 需要导入的数据所在的路径| |-ds|必填参数. 数据源ID| |-s|必选参数. 数据起始时间,导入用户行为数据时指定.格式:YYYY-MM-DD| |-e|必选参数. 数据结束时间,导入用户行为数据时指定.格式:YYYY-MM-DD| |-f|可选参数. 导入数据格式,目前支持JSON,CSV,TSV三种格式.默认值:JSON| |-qf|可选参数. CSV,TSV格式文本限定符.默认值:"| |-sep|可选参数. CSV,TSV格式文本分割符.默认值:,| |-skh|可选参数. CSV,TSV格式设置则自动跳过首行,此参数不需要设置值.| |-attr|可选参数. CSV,TSV格式导入文件的各列按顺序映射到属性名，逗号分隔.userId,event,timestamp必须指定| ## 数据导入之MYSQL 目前支持如下： * 用户属性数据导入 * 用户行为数据导入 ### 使用说明 #### 用户属性数据导入 python3 format_importer.py -m user_variables \ -ds 数据源ID \ -f mysql \ -db_host mysql数据库ip \ -db_user 客户端用户名 \ -db_password 客户端密码 \ -db_port 客户端端口号 \ -sql sql语句 |参数|参数说明| |----|----| |-m|必填参数. 用户属性数据导入-user_variables| |-ds|必填参数. 数据源ID| |-f|必选参数. 导入数据格式,目前支持mysql、hive数据源| |-db_host|必选参数. mysql数据库ip| |-db_user|必选参数. 客户端用户名| |-db_password|必选参数.客户端密码| |-db_port|必选选参数. 客户端端口号| |-sql|必选参数. sql语句| #### 用户行为数据导入 python3 format_importer.py -m events \ -ds 数据源ID \ -f mysql \ -db_host mysql数据库ip \ -db_user 客户端用户名 \ -db_password 客户端密码 \ -db_port 客户端端口号 \ -sql sql语句 \ -s YYYY-MM-DD \ -e YYYY-MM-DD |参数|参数说明| |----|----| |-m|必填参数. 用户行为数据导入-events| |-ds|必填参数. 数据源ID| |-f|必选参数. 导入数据格式,目前支持mysql数据源| |-db_host|必选参数. mysql数据库ip| |-db_user|必选参数. 客户端用户名| |-db_password|必选参数.客户端密码| |-db_port|必选参数. 客户端端口号| |-sql|必选参数. sql语句| |-s|必选参数. 数据起始时间,导入用户行为数据时指定.格式:YYYY-MM-DD| |-e|必选参数. 数据结束时间,导入用户行为数据时指定.格式:YYYY-MM-DD| ## 数据导入之HIVE 目前支持如下： * 用户属性数据导入 * 用户行为数据导入 ### 使用说明 #### 用户属性数据导入 python3 format_importer.py -m user_variables \ -ds 数据源ID \ -f hive \ -db_host hive数据库ip \ -db_user 客户端用户名 \ -db_port 客户端端口号 \ -sql sql语句 |参数|参数说明| |----|----| |-m|必填参数. 用户属性数据导入-user_variables| |-ds|必填参数. 数据源ID| |-f|必选参数. 导入数据格式,目前支持mysql、hive数据源| |-db_host|必选参数. hive数据库ip| |-db_user|必选参数. 客户端用户名| |-db_port|必选选参数. 客户端端口号| |-sql|必选参数. sql语句| #### 用户行为数据导入 python3 format_importer.py -m events \ -ds 数据源ID \ -f hive \ -db_host hive数据库ip \ -db_user 客户端用户名 \ -db_port 客户端端口号 \ -sql sql语句 \ -s YYYY-MM-DD \ -e YYYY-MM-DD |参数|参数说明| |----|----| |-m|必填参数. 用户行为数据导入-events| |-ds|必填参数. 数据源ID| |-f|必选参数. 导入数据格式,目前支持mysql、hive数据源| |-db_host|必选参数. hive数据库ip| |-db_user|必选参数. 客户端用户名| |-db_port|必选参数. 客户端端口号| |-sql|必选参数. sql语句| |-s|必选参数. 数据起始时间,导入用户行为数据时指定.格式:YYYY-MM-DD| |-e|必选参数. 数据结束时间,导入用户行为数据时指定.格式:YYYY-MM-DD|growingio-tools-python-master/importer/__init__.py000066400000000000000000000000001412134335600227720ustar00rootroot00000000000000growingio-tools-python-master/importer/clear_data/000077500000000000000000000000001412134335600227525ustar00rootroot00000000000000growingio-tools-python-master/importer/clear_data/clear_data.py000066400000000000000000000027051412134335600254070ustar00rootroot00000000000000import json import requests from importer.common.config_util import ApiConfig from importer.common.http_util import get_token from importer.common.log_util import logger import tempfile import time import os from importer.meta.data_center import updateClearUserJobStatus, getClearUsers def clearUserData(execute): if str(execute).upper() == 'TRUE': users = getClearUsers() for user in users: assert updateClearUserJobStatus(user) == 'RUNNING' headers = {'Content-Type': 'application/json', 'Authorization': get_token()} ftpPath = '/clearuser/' + str(int(round(time.time() * 1000))) body = {'pipelineDefinitionId': 'ClearUserPipeline', 'identity': 'gio', 'conf': {'$ftpPath': ftpPath}} r = requests.post(ApiConfig.oauth2_uri + '/data-server/scheduler/pipeline/running', headers=headers, data=json.dumps(body)) if r.status_code == 200 and json.loads(r.content)['state'] == 'running': logger.info("提交清理用户删除离线任务") logger.info("待删除用户数据存放FTP位置为:{}".format('/ftp/admin' + ftpPath)) elif str(execute).upper() == 'FALSE': users = getClearUsers() for user in users: assert updateClearUserJobStatus(user) == 'RUNNING' logger.info("等待天任务清理用户删除任务") logger.info("待删除用户数据将存放FTP位置为:{}".format('/ftp/admin/clearuser')) growingio-tools-python-master/importer/clear_user.py000066400000000000000000000021031412134335600233650ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import argparse import os import pathlib import sys sys.path.append(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.meta.meta_create import * from importer.clear_data.clear_data import clearUserData def parse_args(): parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('-m', help='必填参数. 创建事件-clear_users', required=True, metavar="") parser.add_argument('-n','--now', help='可选参数. True-立即执行离线任务,false-天任务执行清理任务', required=True) args = parser.parse_args() return args.__dict__ def main(): args = parse_args() m = args.get("m") if 'clear_users'.__eq__(m): clearUserData(args.get("now")) else: logging.error("请确认填写项目名！") exit(-1) if __name__ == '__main__': main() growingio-tools-python-master/importer/common/000077500000000000000000000000001412134335600221635ustar00rootroot00000000000000growingio-tools-python-master/importer/common/__init__.py000066400000000000000000000000001412134335600242620ustar00rootroot00000000000000growingio-tools-python-master/importer/common/common_util.py000066400000000000000000000057321412134335600250710ustar00rootroot00000000000000import io import pandas as pd import numpy as np import os import shutil from pyhive import hive import pymysql import pytz import datetime import time from hashids import Hashids from pyhive.exc import OperationalError import logging def split_str(s, sep, qualifier): """ 分割复杂字符串 例如：'"sss,ss", userId,event,tismestamp,,"sss,ss",tismes,"sss,ss"' 结果：['sss,ss', 'userId', 'event', 'tismestamp', '', 'sss,ss', 'tismes', 'sss,ss'] :param s: 字符串 :param sep: 分隔符 :param qualifier: 文本限定符 :return: List """ if s is None or s.strip() is '': return [] df = pd.read_csv(io.StringIO(s), sep=sep, quotechar=qualifier, header=None, na_filter=False) return np.array(df.where(df.notnull(), '')).tolist()[0] def get_all_file(path): """ 获取目录下全部文件，包含子文件下的 :param path: 目录或文件路径 :return: [file_abs_path1,file_abs_path2,...] """ res = [] if os.path.isdir(path): for root, dirs, files in os.walk(path, topdown=False): for name in files: res.append(os.path.join(root, name)) else: res.append(path) return res def remove_file(path): """ 删除Path.如果是文件路径直接删除文件；如果是目录，删除目录及目录下全部 :param path: 目录或文件 """ if os.path.isdir(path): shutil.rmtree(path, ignore_errors=True) else: os.rmdir(path) def create_dir(dir_str): """ 创建目录 :param dir_str: 目录 """ if os.path.exists(dir_str) is False: os.makedirs(dir_str) def time_str_to_timestamp_of_tz(time_str, format, timezone): """ 时间字符串转时间戳，带时区 :param time_str: :param format: :param timezone: :return: """ tz = pytz.timezone(timezone) dt = datetime.datetime.strptime(time_str, format) t = tz.localize(dt).astimezone(pytz.utc) return int(time.mktime(t.utctimetuple())) - time.timezone def mysql_connect(user, password, host, port): """ 获取mysql连接 :param user: mysql用户名 :param password: mysql密码 :param host: mysql Host :param port: mysql 端口号 :return: """ return pymysql.connect( host=host, port=port, user=user, charset='utf8', passwd=password, db='information_schema' ) def hive_connect(host, port, user): """ 获取hive连接 :param host: hive Host :param port: hive 端口号 :param user: hive用户名 :return: """ return hive.Connection(host=host, port=port, username=user, database='default') def getVariables(dataCenter): """ 获取变量标识符列表 :param dataCenter: getdataCenterEventVariables、getdataCenterUserVariables :return:标识符列表 """ vars = dataCenter var_list = [] for var in vars: var_list.append(var['key']) return var_list growingio-tools-python-master/importer/common/config_util.py000066400000000000000000000013201412134335600250330ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- import os import pathlib from configparser import ConfigParser config = ConfigParser() config.read(os.path.abspath(str(pathlib.Path(os.path.abspath(__file__)).parent.parent) + "/conf.cfg"), encoding="UTF-8") class ApiConfig: """A ApiConfig Class""" oauth2_uri = config['API']['oauth2_uri'] username = config['API']['username'] password = config['API']['password'] class BaseConfig: """A BaseConfig Class""" timezone = 'Asia/Shanghai' class FTPConfig: host = '' user = '' password = '' port = '' class SaasConfig: uri = config['SAAS']['saas_uri'] token = config['SAAS']['saas_token'] uid = config['SAAS']['saas_uid'] growingio-tools-python-master/importer/common/http_util.py000066400000000000000000000060621412134335600245550ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- import os import subprocess import requests import urllib3 import json import logging from importer.common.config_util import ApiConfig, SaasConfig http = urllib3.PoolManager() def get_token(): """登录获取身份令牌""" cxt = getExtensionClient() payload = {'grant_type': cxt['authorizedGrantTypes'][0], 'client_id': cxt['clientId'], 'client_secret': cxt['clientSecret'], 'scope': cxt['scope'][0], 'username': ApiConfig.username, 'password': ApiConfig.password} headers = {'Content-Type': 'application/x-www-form-urlencoded'} response = requests.post(ApiConfig.oauth2_uri + "/oauth/token", headers=headers, data=payload) if response.status_code == 200: content = json.loads(response.content) return content['access_token'] else: logging.error("获取登录身份令牌失败，请检查配置{}".format(payload)) exit(-1) def send_graphql_post(token, body): """ graphql post请求 :param token: 登录身份令牌 :param body: 请求graphql body :return: 响应数据 """ headers = {'Content-Type': 'application/json', 'Authorization': token} r = requests.post(ApiConfig.oauth2_uri + "/graphql", headers=headers, data=body) if r.status_code == 200: content = json.loads(r.content) if content.keys().__contains__("data") is True: return content['data'] else: logging.error('Graphql请求错误,{}'.format(content['errors'])) exit(-1) else: re = input("Graphql请求超时或失败，是否重新请求，请输入yes/no:") if str(re).lower() == 'yes': send_graphql_post(token, body) else: return def send_rest_get(url, params): """ http post请求 :param url: 请求url :param params: 请求参数 :return: 响应数据 """ headers = {'Content-Type': 'application/json', 'Authorization': SaasConfig.token} # headers = {'Content-Type': 'application/json'} r = requests.get(SaasConfig.uri + url, headers=headers, params=params) if r.status_code == 200: content = json.loads(r.text) return content def put_file_ftp(file_list, target_directory): headers = {'Authorization': 'Bearer ' + get_token()} for file in file_list: file_splits = file.split('/') simple_name = file_splits[len(file_splits) - 1] with open(file, 'rb') as fp: file_data = fp.read() http.request('POST', ApiConfig.oauth2_uri + '/upload', headers=headers, fields={'file': (simple_name, file_data), 'path': target_directory}) def put_file_hdfs(file_list, target_directory): for file in file_list: os.system(f"hdfs dfs -put {file} {target_directory}") def getExtensionClient(): headers = {'Content-Type': 'application/json'} r = requests.get(ApiConfig.oauth2_uri + '/chrome_extension_client', headers=headers) if r.status_code == 200: content = json.loads(r.text) return content growingio-tools-python-master/importer/common/log_util.py000066400000000000000000000015601412134335600243550ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- import logging.config import yaml import os import pathlib path = os.path.abspath(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.common.common_util import create_dir default_log_dir = f'{path}/logs' default_log_file = 'importer.log' default_log_path = f'{default_log_dir}/{default_log_file}' with open(path + '/log_conf.yaml', 'rt') as f: config = yaml.safe_load(f.read()) config_log_path = config['handlers']['file']['filename'] if default_log_file.__eq__(config_log_path): config['handlers']['file']['filename'] = f'{default_log_path}' create_dir(default_log_dir) else: config_log_dir, config_log_file = os.path.split(pathlib.Path(config_log_path)) create_dir(config_log_dir) logging.config.dictConfig(config) logger = logging.getLogger(__name__) growingio-tools-python-master/importer/common/sqlalchemy_util.py000066400000000000000000000151321412134335600257360ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- from sqlalchemy import * from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker from sqlalchemy.sql.functions import now from importer.common.config_util import BaseConfig """ Class Model """ # 创建对象的基类: Base = declarative_base() class UserVariables(Base): """ Table: events.user_variables """ __tablename__ = 'user_variables' id = Column(Integer, primary_key=True) data_center_id = Column(Integer, nullable=False) ai = Column(Text, nullable=False) key = Column(Text, nullable=False) name = Column(Text, nullable=False) description = Column(Text) is_system = Column(Boolean, nullable=False, default=False) status = Column(Text, nullable=False, default='activated') attribution = Column(Text) creator_id = Column(Integer, nullable=False) created_at = Column(TIMESTAMP, nullable=False) updater_id = Column(Integer) updated_at = Column(TIMESTAMP, nullable=False, default=now()) is_high_cardinality = Column(Boolean, default=False) value_type = Column(Text, nullable=False) class CustomEvents(Base): """ Table: events.custom_events """ __tablename__ = 'custom_events' id = Column(Integer, primary_key=True) project_id = Column(Integer, nullable=False) name = Column(Text, nullable=False) description = Column(Text) value_type = Column(Text, nullable=False) key = Column(Text, nullable=False) ai = Column(Text, nullable=False) realtime = Column(Boolean, nullable=False, default=False) business_type = Column(Text) is_system = Column(Boolean, nullable=False, default=False) status = Column(Text, nullable=False, default='activated') creator_id = Column(Integer, nullable=False) created_at = Column(TIMESTAMP, nullable=False) updater_id = Column(Integer) updated_at = Column(TIMESTAMP, nullable=False, default=now()) detected_at = Column(TIMESTAMP) class EventVariables(Base): """ Table: events.event_variables """ __tablename__ = 'event_variables' id = Column(Integer, primary_key=True) data_center_id = Column(Integer, nullable=False) ai = Column(Text, nullable=False) key = Column(Text, nullable=False) name = Column(Text, nullable=False) description = Column(Text) mapping_id = Column(Integer) value_type = Column(Text, nullable=False) is_system = Column(Boolean, nullable=False, default=False) status = Column(Text, nullable=False, default='activated') creator_id = Column(Integer, nullable=False) created_at = Column(TIMESTAMP, nullable=False) updater_id = Column(Integer) updated_at = Column(TIMESTAMP, nullable=False, default=now()) is_high_cardinality = Column(Boolean, default=False) class CustomEventAttributes(Base): """ Table: events.custom_event_attributes """ __tablename__ = 'custom_event_attributes' id = Column(Integer, primary_key=True) data_center_id = Column(Integer, nullable=False) event_id = Column(Integer, nullable=False) attribute_id = Column(Integer, nullable=False) associated_at = Column(TIMESTAMP, nullable=False, default=now()) class CustomEventAttributesView(Base): """ View: events.custom_events LEFT JOIN events.custom_event_attributes LEFT JOIN events.event_variables """ __tablename__ = 'custom_event_attributes_view' id = Column(Integer, primary_key=True) data_center_id = Column(Integer, nullable=False) event_id = Column(Integer, nullable=False) attribute_id = Column(Integer, nullable=False) associated_at = Column(TIMESTAMP, nullable=False, default=now()) class Tunnels(Base): """ Table: cdp.tunnels """ __tablename__ = 'tunnels' id = Column(Integer, primary_key=True) data_center_id = Column(Integer, nullable=False) name = Column(Text, nullable=False) type = Column(Text, nullable=False) config = Column(JSON) status = Column(Text, nullable=False) description = Column(Text, nullable=False) creator_id = Column(Integer, nullable=False) created_at = Column(TIMESTAMP, nullable=False) updater_id = Column(Integer) updated_at = Column(TIMESTAMP, nullable=False, default=now()) key = Column(Text, nullable=False) config_clazz = Column(Text) """ DATABASE: events """ event_engine = create_engine('') event_session = sessionmaker(bind=event_engine)() def get_event_attributes_activated_keys(): """ [event_attributes1,...] """ keys = event_session.query(EventVariables.key).filter(EventVariables.ai == BaseConfig.account_id) \ .filter(EventVariables.status == 'activated') \ .all() result = [] for key in keys: result.append(key[0]) return result def get_custom_event_attributes_view(): """ event_key:[event_attributes1,...] """ custom_event_attributes_view = event_session \ .query(CustomEvents.ai, CustomEvents.id, CustomEvents.key, EventVariables.id, EventVariables.key, EventVariables.value_type) \ .select_from(CustomEvents) \ .outerjoin(CustomEventAttributes, CustomEvents.id == CustomEventAttributes.event_id) \ .outerjoin(EventVariables, EventVariables.id == CustomEventAttributes.attribute_id) \ .filter(CustomEvents.ai == BaseConfig.account_id) \ .filter(CustomEvents.status == 'activated') \ .filter(EventVariables.ai == BaseConfig.account_id) \ .filter(EventVariables.status == 'activated') \ .all() result = {} for view in custom_event_attributes_view: var = result.get(view[2]) if var is None: result[view[2]] = [view[4]] else: var.append(view[4]) return result def get_user_variables_activated_key(): keys = event_session.query(UserVariables.key).filter(UserVariables.ai == BaseConfig.account_id) \ .filter(UserVariables.status == 'activated') \ .all() result = [] for key in keys: result.append(key[0]) return result def count(object, filter): return event_session.query(object).filter(filter).count() def filterone(object, filter): return event_session.query(object).filter(filter).first() """ DATABASE: cdp """ cdp_engine = create_engine('') cdp_session = sessionmaker(bind=cdp_engine)() def get_tunnel_id_by_key(key): """ 获取key对应数据主键 """ ids = cdp_session.query(Tunnels.id, Tunnels.config)\ .filter(Tunnels.key == key)\ .filter(Tunnels.status == 'activated')\ .all() if ids is not None and len(ids) > 0: return ids[0] """ DATABASE: other """ growingio-tools-python-master/importer/conf.cfg000066400000000000000000000007431412134335600223050ustar00rootroot00000000000000# A generic configuration. [API] # CDP页面地址 oauth2_uri = http://127.0.0.1 username = admin password = admin [SAAS] saas_uri = https://www.growingio.com/ # Saas接口请求所需的认证码,具体参考 https://docs.growingio.com/v3/product-manual/projectmange/projectmange/api-token saas_token = Bpu5OhlRsXKyNfycq3jdq5rTUxrtprucxQPOKj2kKAYjfjoJ4AnxcM468ZFDlGg9 # https://www.growingio.com/admin/projects/nxog09md/dashboard 当前项目的UID为 nxog09md saas_uid = GPn0mwq9growingio-tools-python-master/importer/data_import/000077500000000000000000000000001412134335600231765ustar00rootroot00000000000000growingio-tools-python-master/importer/data_import/__init__.py000066400000000000000000000000001412134335600252750ustar00rootroot00000000000000growingio-tools-python-master/importer/data_import/data_events.py000066400000000000000000000327121412134335600260520ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import time import json import tempfile import os from importer.common.http_util import put_file_ftp, put_file_hdfs from importer.data_import.data_model import EventsCv, EventsJson, DataEvent from importer.data_import.data_format_util import * from importer.common import http_util, common_util from json.decoder import JSONDecodeError from importer.common.common_util import get_all_file, remove_file, time_str_to_timestamp_of_tz, getVariables from importer.common.log_util import logger from importer.common.config_util import BaseConfig from importer.meta.data_center import getBindEvent, getdataCenterEventVariables ONE_DAY_MILLISECOND = 86400 * 1000 # 1天(24小时)毫秒值 def events_import(args): """ 用户行为导入，按数据格式处理 """ # Step one: 校验事件数据基础参数，并预处理 # 1. 校验时间 event_start = args.get('event_start') if event_start is None: logger.error("[-s/--event_start]参数值未指定") return event_end = args.get('event_end') if event_end is None: logger.error("[-e/--event_end]参数值未指定") return try: event_start = time_str_to_timestamp_of_tz(event_start, '%Y-%m-%d', BaseConfig.timezone) * 1000 event_end = time_str_to_timestamp_of_tz(event_end, '%Y-%m-%d', BaseConfig.timezone) * 1000 + ONE_DAY_MILLISECOND except TypeError and ValueError: logger.error("[-s/--event_start]或[-e/--event_end]时间参数格式不对,格式为:YYYY-MM-DD") return # 2. 数据源是否为事件 ds = args.get('datasource_id') if 'HISTORY_EVENT' in ds[1]: args['datasource_id'] = ds[1]['HISTORY_EVENT'] else: logger.error("数据源不属于用户行为类型") return # Step one: 按数据格式处理 f = str(args.get('format')) if 'JSON'.__eq__(f): events_import_json( EventsJson(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id'), eventStart=event_start, eventEnd=event_end) ) elif 'CSV'.__eq__(f): events_import_sv( EventsCv(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id'), attributes=args.get('attributes'), separator=",", skipHeader=args.get('skip_header'), eventStart=event_start, eventEnd=event_end) ) elif 'TSV'.__eq__(f): events_import_sv( EventsCv(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id'), attributes=args.get('attributes'), separator='\t', skipHeader=args.get('skip_header'), eventStart=event_start, eventEnd=event_end) ) def events_import_sv(eventsCv): """ 用户行为导入，CSV/TSV格式数据处理 """ # Step 1: 创建临时文件夹，用于存储临时Json文件 current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") try: # Step 2: 校验SV数据，并转为Json格式 n = 0 for path in eventsCv.path: json_file_abs_path = current_tmp_path + '/' + os.path.basename(path).split('.')[0] + '.json' res = sv_import_prepare_process(attributes=eventsCv.attributes, path=path, skip_header=eventsCv.skipHeader, separator=eventsCv.separator, qualifier=eventsCv.qualifier, json_file_abs_path=json_file_abs_path) if res: n = n + 1 # Step 3: 调用Json导入函数:events_import_json if n == len(eventsCv.path): events_import_json( EventsJson(name='events', path=get_all_file(current_tmp_path), debug=eventsCv.debug, format='JSON', eventStart=eventsCv.eventStart, eventEnd=eventsCv.eventEnd, datasourceId=eventsCv.datasourceId) ) finally: # Step 4: 清理Json临时文件 remove_file(current_tmp_path) # SV格式(CSV、TSV) def sv_import_prepare_process(attributes, path, skip_header, separator, qualifier, json_file_abs_path): """ 1.校验数据基本信息 2.SV格式数据转换为Json格式导入 """ # Step 1: 校验有无attributes,有无重复列名 if attributes is None: logger.error("[-attr/--attributes]参数值不存在") return cols = str(attributes).split(',') duplicate_col = check_sv_col_duplicate(cols) if duplicate_col is not None: logger.error(f"[-attr/--attributes]出现重复列值[{duplicate_col}]") return event = getBindEvent() cstm_keys = {} for i in event['dataCenterCustomEvents']: list = [] if len(i['attributes']) != 0: for a in i['attributes']: list.append(a['key']) cstm_keys[i['key']] = list # Json文件临时存储 with open(path, 'r', encoding='utf8') as f: with open(json_file_abs_path, 'w') as wf: line = f.readline().replace('\n', '').replace('\\t', '\t') # Step 2: 校验数据header列是否一致，数量和顺序 if skip_header is True: if check_sv_header_col_count(cols, line.split(separator)) is False: logger.error(f"[-attr/--attributes]参数值列与导入文件[{path}]的列数不一致") return if check_sv_header_col_order(cols, line.split(separator)) is False: logger.error(f"[-attr/--attributes]参数值列与导入文件[{path}]的列顺序不一致") return line = f.readline().replace('\n', '').replace('\\t', '\t') # Step 3: 校验数据列是否一致 while line != '': values = common_util.split_str(line, separator, qualifier) if len(cols) != len(values): logger.error(f"文件[{path}]数据[{line}], 列数与文件头部列数不一致") return # Step 4: 转换为JSON格式 col_value = {} for col, value in tuple(zip(cols, values)): if col != '': col_value[col] = value if col_value['event'] in cstm_keys: attrs = {} for key in col_value: if key in cstm_keys[col_value['event']] and 'userId'.__eq__(key) is False \ and 'event'.__eq__(key) is False and 'userKey'.__eq__(key) is False \ and 'timestamp'.__eq__(key) is False or key.startswith('$') is True: attrs[key] = str(col_value[key]) if 'userKey' in col_value.keys(): data_event = DataEvent(userId=col_value['userId'], event=col_value['event'], userKey=col_value['userKey'], timestamp=col_value['timestamp'], attrs=attrs) else: data_event = DataEvent(userId=col_value['userId'], event=col_value['event'], timestamp=col_value['timestamp'], attrs=attrs) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') else: logger.error( f"文件[{path}]数据[{col_value.__str__()}], " f"自定义事件属性[{str(col_value['event'])}]在GIO平台未定义或未绑定事件属性" ) return line = f.readline().replace('\n', '').replace('\\t', '\t') return True def events_import_json(eventsJson): """ 用户行为导入，Json格式数据处理 """ # Step 1: 执行Debug if eventsJson.debug: if events_debug_process(eventsJson.path, eventsJson.eventStart, eventsJson.eventEnd) is not True: logger.error("Debug校验未通过") return # Step 2: 创建导入任务 job_info = create_task(eventsJson.eventStart, eventsJson.eventEnd, eventsJson.datasourceId) logger.info(f"创建导入任务: {job_info}") # Step 3: 上传数据到FTP # put_file(eventsJson.path, job_info['argument']['directory']) ftp_path = job_info['argument']['directory'] + '/' put_file_ftp(eventsJson.path, ftp_path) # ftp_path = '/ftp/admin' + job_info['argument']['directory'] # put_file_hdfs(eventsJson.path,ftp_path) # Step 4: 启动导入任务 trigger_job(job_info['id']) def trigger_job(id): """ 触发任务 """ body = '''{ "operationName":"executeJob", "variables":{"id":"%s"}, "query":"mutation executeJob($id: HashId!) { executeJob(id: $id) { id stage __typename } }" }''' % id http_util.send_graphql_post(http_util.get_token(), body) def create_task(start, end, tid): """ 创建任务 """ body = '''{ "operationName":"createEventImportJob", "variables":{ "fileType":"ftp", "tunnelId":"%s", "timeRange":"abs:%s,%s" }, "query":"mutation createEventImportJob($tunnelId: HashId!, $timeRange: String, $fileType: String) { createEventImportJob(tunnelId: $tunnelId,timeRange: $timeRange,fileType: $fileType) { id name argument { directory __typename } __typename } }" }''' % (tid, start, end) resp = http_util.send_graphql_post(http_util.get_token(), body) return resp['createEventImportJob'] def events_debug_process(paths, eventStart, eventEnd): """ 用户行为导入Debug 1、校验文件的数据内容合法性, 是否缺失必要字段(userId,event,timestamp) 2、校验文件的数据时间范围合法性 3、校验自定义事件在GIO平台是否定义 """ bind_event = getBindEvent() cstm_keys = {} for i in bind_event['dataCenterCustomEvents']: list = [] if len(i['attributes']) != 0: for a in i['attributes']: list.append(a['key']) cstm_keys[i['key']] = list cstm_attr_keys = getVariables(getdataCenterEventVariables()) count = 0 for path in paths: with open(path, 'r', encoding='utf8') as f: line = f.readline().replace('\n', '') while line.strip() != '': count = count + 1 try: j_dict = json.loads(line) # 1.1 userId if 'userId' not in j_dict: logger.error(f"文件[{path}]数据[{line}], userId不存在") return # 1.2 event if 'event' not in j_dict: logger.error(f"文件[{path}]数据[{line}], event不存在") return # 1.3 timestamp if 'timestamp' not in j_dict: logger.error(f"文件[{path}]数据[{line}], timestamp不存在") return if isinstance(j_dict['timestamp'], int) is False: logger.error(f"文件[{path}]数据[{line}], timestamp数据格式不对") return # 2 timestamp if j_dict['timestamp'] < eventStart or j_dict['timestamp'] >= eventEnd: logger.error(f"文件[{path}]数据[{line}], timestamp时间范围不合法") return # 3 事件 var_keys = cstm_keys.get(j_dict['event']) # 事件key if var_keys is None: logger.error(f"文件[{path}]数据[{line}], 自定义事件[{j_dict['event']}]在GIO平台未定义或未绑定事件属性") return for attr in j_dict['attrs']: # 事件属性 if attr not in cstm_attr_keys and str(attr).startswith('$') is False: logger.error(f"文件[{path}]数据[{line}], 自定义事件属性[{attr}]在GIO平台未定义") return # 事件与属性 if attr not in var_keys and str(attr).startswith('$') is False: logger.error(f"文件[{path}]数据[{line}], 自定义事件属性[{attr}]未与自定义事件[{j_dict['event']}]绑定") return except JSONDecodeError: logger.error(f"文件[{path}]数据[{line}]非JSON格式") return line = f.readline().replace('\n', '') logger.info(f"本次共导入[{count}]行数据") return True growingio-tools-python-master/importer/data_import/data_format_util.py000066400000000000000000000027741412134335600271000ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import ftplib from importer.common.config_util import FTPConfig from collections import Counter def check_sv_header_col_count(attr_header_list, data_header_list): """ 校验CSV/TSV格式数据头-列数 """ if len(attr_header_list) != len(data_header_list): return False def check_sv_header_col_order(attr_header_list, data_header_list): """ 校验CSV/TSV格式数据头-不为''的顺序 """ for i in range(len(attr_header_list)): if attr_header_list[i] != '' and attr_header_list[i] != data_header_list[i]: return False def check_sv_col_duplicate(data_list): """ 校验CSV/TSV格式数据列名是否重复 """ for item in Counter([i for i in data_list if i != '']).items(): if item[1] > 1: return item[0] def put_file(file_list, target_directory): """ FTP上传文件 :param file_list: 待上传文件列表 :param target_directory: 目标目录 :return: """ ftp = ftplib.FTP() ftp.connect(host=FTPConfig.host, port=int(FTPConfig.port)) ftp.login(user=FTPConfig.user, passwd=FTPConfig.password) for file in file_list: file_splits = file.split('/') simple_name = file_splits[len(file_splits) - 1] with open(file, 'rb') as fp: cmd = 'STOR %s/%s' % (target_directory, simple_name) ftp.storbinary(cmd, fp) ftp.close() growingio-tools-python-master/importer/data_import/data_model.py000066400000000000000000000054461412134335600256520ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved class ImportBase: """ Import Base """ def __init__(self, name, path, debug, format, datasourceId): self.name = name self.path = path self.debug = debug self.format = format self.datasourceId = datasourceId class SVBase: """ SV Base """ def __init__(self, skipHeader, separator, qualifier='"'): self.qualifier = qualifier self.separator = separator self.skipHeader = skipHeader class UserVariables(ImportBase): """ 用户属性 """ def __init__(self, name, path, debug, format, datasourceId): ImportBase.__init__(self, name, path, debug, format, datasourceId) class UserVariablesJson(UserVariables): """ 用户属性-Josn格式 """ def __init__(self, name, path, format, datasourceId, debug=True): UserVariables.__init__(self, name, path, debug, format, datasourceId) class UserVariablesSv(UserVariables, SVBase): """ 用户属性-CSV/TSV格式 """ def __init__(self, name, path, debug, format, datasourceId, attributes, skipHeader, separator, qualifier='"'): UserVariables.__init__(self, name, path, debug, format, datasourceId) SVBase.__init__(self, skipHeader, separator, qualifier) self.attributes = attributes class Events(ImportBase): """ 用户行为 """ def __init__(self, name, path, debug, format, datasourceId, eventStart, eventEnd): ImportBase.__init__(self, name, path, debug, format, datasourceId) self.eventStart = eventStart self.eventEnd = eventEnd class EventsJson(Events): """ 用户行为-Josn格式 """ def __init__(self, name, path, format, datasourceId, eventStart, eventEnd, debug=True): Events.__init__(self, name, path, debug, format, datasourceId, eventStart, eventEnd) class EventsCv(Events, SVBase): """ 用户行为-CSV/TSV格式 """ def __init__(self, name, path, debug, format, datasourceId, eventStart, eventEnd, attributes, skipHeader, separator, qualifier='"'): Events.__init__(self, name, path, debug, format, datasourceId, eventStart, eventEnd) SVBase.__init__(self, skipHeader, separator, qualifier) self.attributes = attributes class DataEvent: """ 用户行为大数据规定格式 """ def __init__(self, userId, event, timestamp, attrs, userKey=''): self.userId = userId self.event = event self.userKey = userKey self.timestamp = timestamp self.attrs = attrs class DataUser: """ 用户属性大数据规定格式 """ def __init__(self, userId, attrs): self.userId = userId self.attrs = attrs growingio-tools-python-master/importer/data_import/data_user_variable.py000066400000000000000000000241731412134335600273730ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import time import json import tempfile import os from importer.common.http_util import put_file_ftp, put_file_hdfs from importer.data_import.data_model import UserVariablesSv, UserVariablesJson, DataUser from importer.data_import.data_format_util import * from importer.common import http_util, common_util from importer.common.common_util import get_all_file, remove_file, getVariables from importer.common.log_util import logger from json.decoder import JSONDecodeError from importer.meta.data_center import getdataCenterUserVariables def user_variables_import(args): """ 用户属性导入，按数据格式处理 """ # Step one: 校验事件数据基础参数，并预处理 # 1. 数据源是否为属性 ds = args.get('datasource_id') if 'USER_PROPERTY' in ds[1]: args['datasource_id'] = ds[1]['USER_PROPERTY'] else: logger.error("数据源不属于用户属性类型") return # Step one: 按数据格式处理 f = str(args.get('format')) if 'JSON'.__eq__(f): user_variables_import_json( UserVariablesJson(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id')) ) elif 'CSV'.__eq__(f): user_variables_import_sv( UserVariablesSv(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id'), attributes=args.get('attributes'), separator=",", skipHeader=args.get('skip_header')) ) elif 'TSV'.__eq__(f): user_variables_import_sv( UserVariablesSv(name=args.get('m'), path=args.get('path'), debug=args.get('debug'), format=f, datasourceId=args.get('datasource_id'), attributes=args.get('attributes'), separator='\t', skipHeader=args.get('skip_header')) ) def user_variables_import_sv(userVariablesSv): """ 用户属性导入，CSV/TSV格式数据处理 """ # Step 1: 创建临时文件夹，用于存储临时Json文件 current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") try: # Step 2: 校验SV数据，并转为为Json个数 n = 0 for path in userVariablesSv.path: json_file_abs_path = current_tmp_path + '/' + os.path.basename(path).split('.')[0] + '.json' res = sv_import_prepare_process(attributes=userVariablesSv.attributes, path=path, skip_header=userVariablesSv.skipHeader, separator=userVariablesSv.separator, qualifier=userVariablesSv.qualifier, json_file_abs_path=json_file_abs_path) if res: n = n + 1 # Step 3: 调用Json导入函数:user_variables_import_json if n == len(userVariablesSv.path): user_variables_import_json( UserVariablesJson(name='user_variables', path=get_all_file(current_tmp_path), debug=userVariablesSv.debug, format='JSON', datasourceId=userVariablesSv.datasourceId) ) finally: # Step 4: 清理Json临时文件 remove_file(current_tmp_path) # SV格式(CSV、TSV) def sv_import_prepare_process(attributes, path, skip_header, separator, qualifier, json_file_abs_path): """ 1.校验数据基本信息 2.SV格式数据转换为Json格式导入 """ # Step 1: 校验有无attributes,有无重复列名 if attributes is None: logger.error("[-attr/--attributes]参数值不存在") return cols = str(attributes).split(',') duplicate_col = check_sv_col_duplicate(cols) if duplicate_col is not None: logger.error(f"[-attr/--attributes]出现重复列值[{duplicate_col}]") return keys = getVariables(getdataCenterUserVariables()) with open(path, 'r', encoding='utf8') as f: with open(json_file_abs_path, 'w') as wf: line = f.readline().replace('\n', '').replace('\\t', '\t') # Step 2: 校验数据header列是否一致，数量和顺序 if skip_header is True: if check_sv_header_col_count(cols, line.split(separator)) is False: logger.error(f"[-attr/--attributes]参数值列与导入文件[{path}]的列数不一致") return if check_sv_header_col_order(cols, line.split(separator)) is False: logger.error(f"[-attr/--attributes]参数值列与导入文件[{path}]的列顺序不一致") return line = f.readline().replace('\n', '').replace('\\t', '\t') # Step 3: 校验数据列是否一致 while line != '': values = common_util.split_str(line, separator, qualifier) if len(cols) != len(values): logger.error(f"文件[{path}]数据[{line}]列数与文件头部列数不一致") return # Step 4: 转换为JSON格式 col_value = {} for col, value in tuple(zip(cols, values)): if col != '': col_value[col] = str(value) attrs = {} for key, value in col_value.items(): if len(str(value)) != 0: if key.startswith('$') is False and key not in keys \ and 'userId'.__eq__(key) is False: logger.error(f"文件[{path}]数据[{line}]用户属性[{key}]在GIO平台未定义") return elif 'userId'.__eq__(key) is False: if col_value[key] != ' ' and col_value[key] != '\\N' and col_value[key] != '\\n': attrs[key] = col_value[key] data_event = DataUser(userId=col_value['userId'], attrs=attrs) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') line = f.readline().replace('\n', '').replace('\\t', '\t') return True def user_variables_import_json(userVariablesJson): """ 用户属性导入，Json格式数据处理 """ # Step 1: 执行Debug if userVariablesJson.debug: if user_variables_debug_process(userVariablesJson.path) is not True: logger.error("Debug校验未通过") return # Step 2: 创建导入任务 job_info = create_task(userVariablesJson.datasourceId) logger.info(f"创建导入任务: {job_info}") # Step 3: 上传数据到FTP # put_file(userVariablesJson.path, job_info['argument']['directory']) ftp_path = job_info['argument']['directory'] + '/' put_file_ftp(userVariablesJson.path, ftp_path) # ftp_path = '/ftp/admin' + job_info['argument']['directory'] # put_file_hdfs(userVariablesJson.path,ftp_path) # Step 4: 启动导入任务 trigger_job(job_info['id']) def trigger_job(id): body = '''{ "operationName":"executeJob", "variables":{"id":"%s"}, "query":"mutation executeJob($id: HashId!) { executeJob(id: $id) { id stage __typename } }" }''' % id http_util.send_graphql_post(http_util.get_token(), body) def create_task(ds): body = '''{ "operationName": "createEventImportJob", "variables": { "fileType":"ftp", "timeRange":"", "tunnelId": "%s" }, "query": "mutation createEventImportJob($tunnelId: HashId!, $timeRange: String, $fileType: String) { createEventImportJob(tunnelId: $tunnelId, timeRange: $timeRange,fileType: $fileType) { id name argument { directory __typename } __typename } }" }''' % ds resp = http_util.send_graphql_post(http_util.get_token(), body) return resp['createEventImportJob'] def user_variables_debug_process(paths): """ 用户属性导入Debug 1、校验有无userId 2、校验用户属性(条件:是否是平台内置和是否定义) """ keys = getVariables(getdataCenterUserVariables()) count = 0 for path in paths: with open(path, 'r', encoding='utf8') as f: line = f.readline().replace('\n', '') while line.strip() != '': count = count + 1 try: j_dict = json.loads(line) # userId if 'userId' not in j_dict: logger.error(f"文件[{path}]数据[{line}]userId不存在") return # 用户属性 if isinstance(j_dict['attrs'], dict) is False: logger.error(f"文件[{path}]数据[{line}]attrs数据格式不对") return for key in j_dict['attrs']: if j_dict['attrs'][key] is None: logger.error(f"文件[{path}]数据[{line}], 用户属性[{key}]的值为NULL,请检查原始数据") return if key.startswith('$') is False and key not in keys: logger.error(f"文件[{path}]数据[{line}], 用户属性[{key}]在GIO平台未定义") return except JSONDecodeError: logger.error(f"文件[{path}]数据[{line}]非JSON格式") return line = f.readline().replace('\n', '') logger.info(f"本次共导入[{count}]行数据") return True growingio-tools-python-master/importer/data_importer.py000066400000000000000000000113001412134335600240720ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved """ 用户属性 """ import argparse import os import pathlib import sys sys.path.append(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.data_import.data_events import events_import from importer.data_import.data_user_variable import user_variables_import from importer.common.common_util import get_all_file from importer.common.log_util import logger from importer.meta.data_center import getTunnels def parse_args(): parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('-m', help='必填参数. 用户属性数据导入:user_variables，用户行为数据导入:events.', required=True, default='', type=str) parser.add_argument('-p', '--path', help='必填参数. 需要导入的数据所在的路径', required=True, default='', type=str) parser.add_argument('-ds', '--datasource_id', help='必填参数. 数据源ID.', required=True, default='', type=str) parser.add_argument('-f', '--format', help='可选参数. 导入数据格式,目前支持JSON,CSV,TSV三种格式.', default='JSON', type=str) # 暂时取消 # parser.add_argument('-d', '--debug', # help='可选参数. True-导入数据全量校验但不导,false-立即创建导入任务进入导入队列.', # default=True) parser.add_argument('-s', '--event_start', help='可选参数. 数据起始时间,导入用户行为数据时指定', default='', type=str) parser.add_argument('-e', '--event_end', help='可选参数. 数据结束时间,导入用户行为数据时指定', default='', type=str) parser.add_argument('-qf', '--qualifier', help='可选参数. 文本限定符.', default='"', type=str) parser.add_argument('-sep', '--separator', help='可选参数. 文本分割符.', default=',', type=str) parser.add_argument('-skh', '--skip_header', help='可选参数. 设置则自动跳过首行.', action='store_true') parser.add_argument('-attr', '--attributes', help='可选参数. 导入文件的各列按顺序映射到属性名，英文逗号分隔.', default='', type=str) parser.add_argument('-v', '--version', action='version', version='Gio_DataImporter_1.0') args = parser.parse_args() return args.__dict__ def main(): # Step one: 解析命令行参数 args = parse_args() logger.debug(f"解析命令行参数: {args}") # Step one: 校验基础参数,并预处理参数 # 1. 校验导入文件 p = args.get('path') ps = [] if os.path.exists(p): ps.extend(get_all_file(p)) else: logger.error("需要导入的数据文件不存在") return args['path'] = ps # 2. 校验Debug参数 # 暂时取下 # d = str(args.get('debug')).upper() # if 'TRUE'.__eq__(d): # d = True # elif 'FALSE'.__eq__(d): # d = False # else: # logger.error("[-d/--debug]参数值不对") # return args['debug'] = 'TRUE' # 3. 校验数据格式 f = str(args.get('format')).upper() if 'JSON'.__eq__(f) is False and 'CSV'.__eq__(f) is False and 'TSV'.__eq__(f) is False: logger.error("目前支持JSON,CSV,TSV三种格式") return args['format'] = f # 4. 校验数据源 tunnels = getTunnels() if args.get('datasource_id') not in tunnels: logger.error("数据源不存在") return args['datasource_id'] = [args.get('datasource_id'), tunnels[args.get('datasource_id')]] # Step three: 按导入模块处理 m = args.get('m') if 'user_variables'.__eq__(m): user_variables_import(args) elif 'events'.__eq__(m): events_import(args) else: logger.warn("目前只支持用户行为数据和用户属性数据导入") return if __name__ == '__main__': logger.info("Data Importer Start") main() logger.info("Data Importer Finish") growingio-tools-python-master/importer/db_import/000077500000000000000000000000001412134335600226525ustar00rootroot00000000000000growingio-tools-python-master/importer/db_import/database_import.py000066400000000000000000000035571412134335600263740ustar00rootroot00000000000000import json import os import tempfile import time from importer.common.common_util import time_str_to_timestamp_of_tz from importer.common.config_util import BaseConfig from importer.common.log_util import logger from importer.data_import.data_events import ONE_DAY_MILLISECOND, events_import_json from importer.db_import.hive_import import event_hive_import, user_hive_import from importer.db_import.mysql_import import event_mysql_import, user_mysql_import def events(args): try: start = time_str_to_timestamp_of_tz(args.get('start_time'), '%Y-%m-%d', BaseConfig.timezone) * 1000 end = time_str_to_timestamp_of_tz(args.get('end_time'), '%Y-%m-%d', BaseConfig.timezone) * 1000 + ONE_DAY_MILLISECOND except TypeError and ValueError: logger.error("[-s/--start_time]或[-e/--end_time]时间参数格式不对,格式为:YYYY-MM-DD") exit(-1) ds = args.get('datasource_id') if 'HISTORY_EVENT' in ds[1]: args['datasource_id'] = ds[1]['HISTORY_EVENT'] else: logger.error("数据源不属于用户行为类型") exit(-1) f = args.get('format') if 'MYSQL'.__eq__(str(f).upper()): event_mysql_import(args, start, end) elif 'HIVE'.__eq__(str(f).upper()): event_hive_import(args, start, end) else: logger.error("format不在取值范围内") exit(-1) def user_variables(args): ds = args.get('datasource_id') if 'USER_PROPERTY' in ds[1]: args['datasource_id'] = ds[1]['USER_PROPERTY'] else: logger.error("数据源不属于用户属性类型") exit(-1) if 'MYSQL'.__eq__(str(args.get('format')).upper()): user_mysql_import(args) elif 'HIVE'.__eq__(str(args.get('format')).upper()): user_hive_import(args) else: logger.error("format不在取值范围内") exit(-1) growingio-tools-python-master/importer/db_import/hive_import.py000066400000000000000000000155431412134335600255610ustar00rootroot00000000000000import json import os import tempfile import time from pyhive.exc import OperationalError from importer.common.common_util import get_all_file, remove_file, hive_connect, getVariables from importer.common.log_util import logger from importer.data_import.data_events import events_import_json from importer.data_import.data_model import EventsJson, DataEvent, DataUser, UserVariablesJson from importer.data_import.data_user_variable import user_variables_import_json from importer.meta.data_center import getBindEvent, getdataCenterUserVariables def event_hive_import(args, start, end): """ 行为数据导入，HIVE数据源 """ try: conn = hive_connect(host=args.get('host'), port=int(args.get('port')), user=args.get('user') ) except TypeError: logger.error("HIVE连接失败。") exit(-1) cursor = conn.cursor() try: sql = args.get('sql') cursor.execute(sql) logger.info(sql) except OperationalError: logger.error("请检查SQL语句") exit(-1) desc = cursor.description desc_list = [] for d in desc: desc_list.append(d[0]) if 'userId' not in desc_list or 'event' not in desc_list or 'timestamp' not in desc_list: logger.error("userId或event或timestamp字段不存在") exit(-1) current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") json_file_abs_path = current_tmp_path + '/' + 'tmp_events.json' event = getBindEvent() res = {} for i in event['dataCenterCustomEvents']: list = [] for a in i['attributes']: list.append(a['key']) res[i['key']] = list try: wf = open(json_file_abs_path, 'w') cnt = 0 while True: cnt += 1 data = cursor.fetchmany(size=args.get('batch')) if len(data) == 0 and cnt == 1: logger.error("数据为空") exit(-1) elif len(data) == 0 and cnt > 1: events_import_json( EventsJson(name='events', path=get_all_file(current_tmp_path), format='JSON', debug=False, eventStart=start, eventEnd=end, datasourceId=args.get('datasource_id')) ) break else: json_list = [] for row in data: res = {} var = {} for a in range(len(row)): if desc_list[a] == 'userId' or desc_list[a] == 'event' or desc_list[a] == 'timestamp': res[desc_list[a]] = row[a] else: var[desc_list[a]] = row[a] res['attrs'] = var json_list.append(res) for json_str in json_list: if json_str['event'] in res: for key in json_str['attrs']: if key not in res[json_str['event']]: logger.error(f"自定义事件属性[{key}]在GIO平台未定义或未绑定事件属性") exit(-1) data_event = DataEvent(userId=json_str['userId'], event=json_str['event'], timestamp=json_str['timestamp'], attrs=json_str['attrs']) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') else: logger.error(f"事件[{json_str['event']}]在GIO平台未定义或者未绑定属性") exit(-1) finally: remove_file(current_tmp_path) def user_hive_import(args): """ 用户属性导入，HIVE数据源 """ try: conn = hive_connect(host=args.get('host'), port=int(args.get('port')), user=args.get('user')) except TypeError: logger.error("HIVE连接失败。") exit(-1) cursor = conn.cursor() try: sql = args.get('sql') cursor.execute(sql) logger.info(sql) except OperationalError: logger.error("请检查SQL语句") exit(-1) desc = cursor.description desc_list = [] for d in desc: desc_list.append(d[0]) if 'userId' not in desc_list: logger.error("userId字段不存在") exit(-1) current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") keys = getVariables(getdataCenterUserVariables()) json_file_abs_path = current_tmp_path + '/' + 'tmp_user.json' try: wf = open(json_file_abs_path, 'w') cnt = 0 while True: cnt += 1 data = cursor.fetchmany(size=args.get('batch')) if len(data) == 0 and cnt == 1: logger.error("数据为空") exit(-1) elif len(data) == 0 and cnt > 1: user_variables_import_json( UserVariablesJson(name='user_variables', path=get_all_file(current_tmp_path), debug=False, format='JSON', datasourceId=args.get('datasource_id')) ) break else: json_list = [] for row in data: res = {} var = {} for a in range(len(row)): if desc_list[a] == 'userId': res[desc_list[a]] = row[a] else: var[desc_list[a]] = row[a] res['attrs'] = var json_list.append(res) for json_str in json_list: for key in json_str['attrs']: if key not in keys and key.startswith("$") is False: logger.error("用户属性{}在GIO平台未定义".format(key)) exit(-1) data_event = DataUser(userId=json_str['userId'], attrs=json_str['attrs']) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') finally: remove_file(current_tmp_path) growingio-tools-python-master/importer/db_import/mysql_import.py000066400000000000000000000141531412134335600257670ustar00rootroot00000000000000import json import os import tempfile import time from pymysql import OperationalError, MySQLError from importer.common.common_util import mysql_connect, get_all_file, remove_file, getVariables from importer.common.log_util import logger from importer.data_import.data_events import events_import_json from importer.data_import.data_model import EventsJson, DataEvent, DataUser, UserVariablesJson from importer.data_import.data_user_variable import user_variables_import_json from importer.meta.data_center import getBindEvent, getdataCenterUserVariables def event_mysql_import(args, start, end): """ 行为数据导入，MYSQL数据源 """ try: conn = mysql_connect(user=args.get('db_user'), password=args.get('db_password'), host=args.get('db_host'), port=int(args.get('db_port'))) except MySQLError and OperationalError: logger.error("MYSQL连接失败。") exit(-1) cursor = conn.cursor() try: sql = args.get('sql') cursor.execute(sql) logger.info(sql) except SyntaxError or MySQLError or OperationalError: logger.error("请检查SQL语句") exit(-1) desc = cursor.description desc_list = [] for d in desc: desc_list.append(d[0]) if 'userId' not in desc_list or 'event' not in desc_list or 'timestamp' not in desc_list: logger.error("userId或event或timestamp字段不存在") exit(-1) data = cursor.fetchall() if len(data) == 0: logger.error("数据为空") exit(-1) json_list = [] for row in data: res = {} var = {} for a in range(len(row)): if desc_list[a] == 'userId' or desc_list[a] == 'event' or desc_list[a] == 'timestamp': res[desc_list[a]] = row[a] else: var[desc_list[a]] = row[a] res['attrs'] = var json_list.append(res) current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") json_file_abs_path = current_tmp_path + '/' + 'tmp_events.json' event = getBindEvent() res = {} for i in event['dataCenterCustomEvents']: list = [] for a in i['attributes']: list.append(a['key']) res[i['key']] = list try: with open(json_file_abs_path, 'w') as wf: for json_str in json_list: if json_str['event'] in res: for key in json_str['attrs']: if key not in res[json_str['event']]: logger.error(f"自定义事件属性[{key}]在GIO平台未定义或未绑定事件属性") exit(-1) data_event = DataEvent(userId=json_str['userId'], event=json_str['event'], timestamp=json_str['timestamp'], attrs=json_str['attrs']) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') else: logger.error(f"事件[{json_str['event']}]在GIO平台未定义或者未绑定属性") exit(-1) events_import_json( EventsJson(name='events', path=get_all_file(current_tmp_path), format='JSON', debug=False, eventStart=start, eventEnd=end, datasourceId=args.get('datasource_id')) ) finally: remove_file(current_tmp_path) def user_mysql_import(args): """ 用户属性导入，MYSQL数据源 """ try: conn = mysql_connect(user=args.get('db_user'), password=args.get('db_password'), host=args.get('db_host'), port=int(args.get('db_port'))) except MySQLError and OperationalError: logger.error("MYSQL连接失败。") exit(-1) cursor = conn.cursor() try: sql = args.get('sql') cursor.execute(sql) logger.info(sql) except SyntaxError or MySQLError or OperationalError: logger.error("请检查SQL语句") exit(-1) desc = cursor.description desc_list = [] for d in desc: desc_list.append(d[0]) if 'userId' not in desc_list: logger.error("userId字段不存在") exit(-1) data = cursor.fetchall() if len(data) == 0: logger.error("数据为空") exit(-1) json_list = [] for row in data: res = {} var = {} for a in range(len(row)): if desc_list[a] == 'userId': res[desc_list[a]] = row[a] else: var[desc_list[a]] = row[a] res['attrs'] = var json_list.append(res) current_tmp_path = tempfile.gettempdir() + '/' + str(int(round(time.time() * 1000))) if os.path.exists(current_tmp_path) is False: os.makedirs(current_tmp_path) logger.debug(f"临时存储Json文件目录：[{current_tmp_path}]") keys = getVariables(getdataCenterUserVariables()) json_file_abs_path = current_tmp_path + '/' + 'tmp_user.json' try: with open(json_file_abs_path, 'w') as wf: for json_str in json_list: for key in json_str['attrs']: if key not in keys and key.startswith("$") is False: logger.error("用户属性{}在GIO平台未定义".format(key)) exit(-1) data_event = DataUser(userId=json_str['userId'], attrs=json_str['attrs']) wf.write(json.dumps(data_event.__dict__, ensure_ascii=False)) wf.write('\n') user_variables_import_json( UserVariablesJson(name='user_variables', path=get_all_file(current_tmp_path), debug=False, format='JSON', datasourceId=args.get('datasource_id')) ) finally: remove_file(current_tmp_path) growingio-tools-python-master/importer/example/000077500000000000000000000000001412134335600223265ustar00rootroot00000000000000growingio-tools-python-master/importer/example/__init__.py000066400000000000000000000000001412134335600244250ustar00rootroot00000000000000growingio-tools-python-master/importer/format_importer.py000066400000000000000000000057351412134335600244700ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import argparse import os import pathlib import sys import logging sys.path.append(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.db_import.database_import import events, user_variables from importer.common.log_util import logger from importer.meta.meta_create import * def parse_args(): parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('-m', help='必填参数. 事件events，用户属性user_variables', required=True, type=str) parser.add_argument('-ds', '--datasource_id', help='必填参数. 数据源ID.', required=True, type=str) parser.add_argument('-f', '--format', help='可选参数. 导入数据格式,目前支持mysql、hive格式.', type=str, metavar="") parser.add_argument('-db_host', '--host', help='必填参数. 客户数据源地址', metavar="") parser.add_argument('-db_user', '--user', help='必填参数. 客户数据源用户', type=str, metavar="") parser.add_argument('-db_pass', '--password', help='必填参数. 客户数据源密码', type=str, metavar="") parser.add_argument('-db_port', '--port', help='必填参数. 客户数据源端口', metavar="") parser.add_argument('-sql', '--sql', help='必选参数. sql语句', type=str, metavar="") parser.add_argument('-b', '--batch', help='可选参数. hive模式下每批次处理多少条数据', default=100000, metavar="") parser.add_argument('-s', '--start_time', help='用户行为数据必选参数. 开始时间', metavar="") parser.add_argument('-e', '--end_time', help='用户行为数据必选参数. 结束时间', metavar="") args = parser.parse_args() return args.__dict__ def main(): args = parse_args() tunnels = getTunnels() if args.get('datasource_id') not in tunnels: logger.error("数据源不存在") return m = args.get("m") args['datasource_id'] = [args.get('datasource_id'), tunnels[args.get('datasource_id')]] if 'events'.__eq__(m): events(args) elif 'user_variables'.__eq__(m): user_variables(args) else: logging.error("请确认填写项目名！") exit(-1) if __name__ == '__main__': main() growingio-tools-python-master/importer/init.sh000066400000000000000000000003501412134335600221700ustar00rootroot00000000000000#!/bin/bash yum install -y gcc-c++ yum install -y libevent-devel yum install -y python3-devel yum install -y python-devel yum install -y gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64 python3 -m pip install -r requirements.txtgrowingio-tools-python-master/importer/log_conf.yaml000066400000000000000000000007311412134335600233460ustar00rootroot00000000000000root: level: DEBUG handlers: [file,console] version: 1 disable_existing_loggers: False formatters: simple: format: "%(asctime)s %(levelname)-8s [%(threadName)s] %(name)s %(filename)s: %(lineno)d: %(message)s" handlers: console: class: logging.StreamHandler formatter: simple file: class: logging.handlers.TimedRotatingFileHandler filename: importer.log formatter: simple encoding: UTF-8 when: d interval: 1 backupCount: 30 growingio-tools-python-master/importer/meta/000077500000000000000000000000001412134335600216215ustar00rootroot00000000000000growingio-tools-python-master/importer/meta/__init__.py000066400000000000000000000000001412134335600237200ustar00rootroot00000000000000growingio-tools-python-master/importer/meta/check_util.py000066400000000000000000000143521412134335600243120ustar00rootroot00000000000000import logging from importer.meta.data_center import * import re def check_key(str): """ 校验标识符KEY，不能为空，长度小于50，只可以由字母数字下划线组成，且只能字母开头 :param str: 字符串 :return: """ if (str == ''): logging.error("事件名不能为空") exit(-1) elif (len(str) > 50): logging.error("事件名<{}>长度超限".format(str)) exit(-1) elif (str[0].isdigit() or is_chinese(str) or match(str)): logging.error("事件名<{}>不合法，只可以由字母数字下划线组成，且只能字母开头~".format(str)) exit(-1) def check_name(name, key): """ 校验name，长度小于50，当name为空，则默认与标识符一致 :param name: 名称 :param key: 标识符 :return: """ if name != "" and len(name) > 50: logging.error("显示名<{}>长度超限~".format(name)) exit(-1) elif name == "": return key # elif isexsit_eventvar(name) or isexsit_usertvar(name): # logging.error("名称<{}>已重复~".format(name)) # exit(-1) else: return name def isexsit_eventvar(name): """ 校验事件属性名称是否存在 :param name: 名称 :return: True/False """ event_variables = getdataCenterEventVariables() for var in event_variables: if var['name'] == name: return True return False def isexsit_usertvar(name): """ 校验用户属性名称是否存在 :param name: 名称 :return: True/False """ user_variables = getdataCenterUserVariables() for var in user_variables: if var['name'] == name: return True return False def check_event_valuetype(tp): """ 校验事件数据类型 :param tp: 数据类型 :return: """ if tp == '': logging.error("数据类型<{}>未设置".format(tp)) exit(-1) elif tp not in ('string', 'int', 'double'): logging.error("数据类型<{}>不在可选值范围".format(tp)) exit(-1) def check_user_valuetype(tp): """ 校验用户数据类型 :param tp: 数据类型 :return: """ if tp == '': logging.error("数据类型<{}>未设置".format(tp)) exit(-1) elif tp not in ('string', 'int', 'date'): logging.error("数据类型<{}>不在可选值范围".format(tp)) exit(-1) def check_event_exsit(token, key): """ 校验事件标识符KEY是否存在 :param token: 登录令牌 :param key: 标识符 :return: """ custom_events = getdataCenterCustomEvents(token) for var in custom_events: if var['key'] == key: return True, var else: return False, '' def check_event_attr_exsit(token, key): """ 校验事件标识符KEY是否存在 :param token: 登录令牌 :param key: 标识符 :return: """ custom_events = getdataCenterCustomEventsAndAttr(token) id_list = [] for var in custom_events: if var['key'] == key: if len(var['attributes']) != 0: for a in var['attributes']: id_list.append(a['id']) return True, var, id_list else: return True, var, id_list else: return False, '', [] def check_event_var_exsit(token, key): """ 校验事件属性标识符KEY是否存在 :param token: 登录令牌 :param key: 标识符 :return: """ event_variables = getdataCenterEventVariables(token) for var in event_variables: if var['key'] == key: return True, var else: return False, '' def check_user_var_exsit(token, key): """ 校验用户属性标识符KEY是否存在 :param token: 登录令牌 :param key: 标识符 :return: """ user_variables = getdataCenterUserVariables(token) for var in user_variables: if var['key'] == key: return True, var else: return False, '' def check_bind_event_exsit(token, key): """ 校验事件与事件属性 :param token: 登录令牌 :param key: 标识符 :return: """ custom_events = getdataCenterCustomEvents(token) for var in custom_events: if var['key'] == key: return True, var['id'], var['description'] else: return False, '', '' def check_key_name(var_list, key, name): """ 校验标识符、名称 :param var_list: 变量列表 :param key: 标识符 :param name: 名称 :return: """ for var in var_list: if var['key'] == key: logging.error("标识符<{}>已存在~".format(key)) exit(-1) elif var['name'] == name: logging.error("名称<{}>已存在~".format(name)) exit(-1) def is_chinese(string): """ 检测字符串是否存在汉字 :param string: 字符串 :return:True/False """ for ch in string: if u'\u4e00' <= ch <= u'\u9fff': return True return False def match(str): """ 只可以由字母数字下划线组成，且只能字母开头 :param str: 字符串 :return:True/False """ mh = re.match(r'[0-9a-zA-Z_]*', str) if len(mh.group()) != len(str): return True else: return False def check_attr(token, attr): """ 校验事件与事件属性 :param token: 登录令牌 :param attr: 事件属性 :return:id_list,key_list """ attrs = str(attr).split(",") key_list = [] id_list = [] error_list = [] event_variables = getdataCenterEventVariables(token) for a in attrs: for event in event_variables: if event['key'] == a: id_list.append(event['id']) key_list.append(event['key']) break else: error_list.append(a) if len(error_list) == len(attrs): logging.error("绑定事件属性失败，事件属性:{}不存在,请先创建事件属性~".format(error_list)) exit(-1) elif len(error_list) != 0: logging.error("事件属性:{}不存在,请先创建事件属性~".format(error_list)) return id_list, key_list growingio-tools-python-master/importer/meta/data_center.py000066400000000000000000000127171412134335600244540ustar00rootroot00000000000000import json from importer.common.http_util import send_graphql_post, get_token # 获取custom_events表(id,key,name,description)数据 def getdataCenterCustomEvents(token=get_token()): """ 获取 CustomEvents (id,key,name,description) """ body = """{"query":"query MyQuery { dataCenterCustomEvents { id key name description}}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content['dataCenterCustomEvents'] return data_info def getdataCenterCustomEventsAndAttr(token=get_token()): """ 获取 CustomEvents (id,key,name,description,attributes(id,key)) """ body = """{"query":"query MyQuery { dataCenterCustomEvents { id key name description attributes {id key}}}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content['dataCenterCustomEvents'] return data_info def getdataCenterUserVariables(token=get_token()): """ 获取 UserVariables (id,key,name,description,valueType) """ body = """{"query":"query MyQuery { dataCenterUserVariables { id key name description valueType}}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content['dataCenterUserVariables'] return data_info def getdataCenterEventVariables(token=get_token()): """ 获取 EventVariables (id,key,name,description,valueType) """ body = """{"query":"query MyQuery {dataCenterEventVariables { id key name description valueType}}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content['dataCenterEventVariables'] return data_info # 获取元数据 def getMetaData(token=get_token()): """ 获取元数据 """ body = """{"query":"query MyQuery {dataCenterCustomEvents { key name description } dataCenterEventVariables { key name valueType description } dataCenterUserVariables { key name valueType description }}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content return data_info def getBindEvent(token=get_token()): """ 获取绑定事件和关联事件属性 """ body = """{"query":"query MyQuery { dataCenterCustomEvents { key attributes { key } }}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) data_info = content return data_info def getTunnels(token=get_token()): """ 获取 tunnels(id key type config { ... on FileTunnelConfig { type } }) """ body = """{"query":"query MyQuery { tunnels {id key type config { ... on FileTunnelConfig { type } } }}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) tunnels = content['tunnels'] t_res = {} for t in tunnels: res = {} if t['type'] == 'FILE': res[t['config']['type']] = t['id'] t_res[t['key']] = res return t_res def getClearUsers(token=get_token()): """ 获取待删除用户列表 """ body = """{"query":"query MyQuery {clearUsers {id userId stage}}","variables":null,"operationName":"MyQuery"}""" content = send_graphql_post(token, body) clearusers = content['clearUsers'] # print(clearusers) res = {} for user in clearusers: if(user['stage']=='WAITING'): res[user['id']] = user['userId'] return res def getGidByUserId(userid, token=get_token()): """ 获取用户ID对应Gid """ body = """{ "operationName":"getGidByUser", "variables":{ "userId":"%s" }, "query":"query getGidByUser($userId: String!) { getGidByUser(userId: $userId) } "} """ % (userid) content = send_graphql_post(token, body) return content['getGidByUser'] def getGidAllData(gid, token=get_token()): """ 获取Gid下所有埋点、用户属性数据 """ body = """{ "operationName":"getAllUserData", "variables":{ "gioId":"%s" }, "query":"query getAllUserData($gioId: Long!) { getAllUserData(gioId: $gioId) } "} """ % (gid) content = send_graphql_post(token, body) return content['getAllUserData'] def updateClearUserJobStatus(id, token=get_token()): """ 更新待删除用户状态 """ body = """{ "operationName":"updateClearUser", "variables":{ "id":"%s", "clearUser":{ "stage":"RUNNING" } }, "query":"mutation updateClearUser($id: HashId!,$clearUser:ClearUserInput!) { updateClearUser(id:$id,clearUser:$clearUser) { stage } } "} """ % (id) content = send_graphql_post(token, body) return content['updateClearUser']['stage'] def deleteClearUserData(gid, token=get_token()): """ 删除Gid下埋点、用户属性数据 """ body ="""{ "operationName":"deleteClearUserData", "variables":{ "gioId":"%s" }, "query":"mutation deleteClearUserData($gioId: Long!) { deleteClearUserData(gioId: $gioId) } "} """%(gid) content = send_graphql_post(token, body) return content['deleteClearUserData']growingio-tools-python-master/importer/meta/meta_create.py000066400000000000000000000275311412134335600244540ustar00rootroot00000000000000import json from importer.common.http_util import send_graphql_post, get_token from importer.meta.check_util import * from importer.meta.data_center import getMetaData, getBindEvent """ 元数据创建 """ def create_event(token, key, name, desc): """ 创建事件 """ names = check_name(name, key) flag, var, id_list = check_event_attr_exsit(token, key) str_sub = '' if len(id_list) != 0: attr_str = '' for i in range(len(id_list)): attr_str += '{\"id\":\"' + str(id_list[i]) + '\"},' str_sub = attr_str[:len(attr_str) - 1] if flag: if desc == '' and name != '': body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s", "valueType":"counter","attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], names, key, var['description'], str_sub) elif names == key and desc != '': body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s", "valueType":"counter","attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], var['name'], key, desc, str_sub) elif name != '' and desc != '': body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s", "valueType":"counter","attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], names, key, desc, str_sub) else: body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s", "valueType":"counter","attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], var['name'], key, var['description'], str_sub) else: body = """{"operationName":"createDataCenterCustomEvent","variables":{"customEvent":{"name":"%s","key":"%s","valueType":"counter", "description":"%s","attributes":[]}},"query":"mutation createDataCenterCustomEvent($customEvent: CustomEventInput!) {createDataCenterCustomEvent(customEvent: $customEvent) {name __typename}}"}""" % (names, key, desc) content = send_graphql_post(token, body.encode('utf-8')) return content # 创建事件属性 def create_event_variables(token, key, valuetype, name, desc): """ 创建事件属性 """ name = check_name(name, key) check_event_valuetype(valuetype) flag, var = check_event_var_exsit(token, key) if flag: if desc == '': body = """{"operationName":"updateDataCenterEventVariable","variables":{"id":"%s","eventVariable":{"name":"%s","key":"%s", "description":"%s","valueType":"%s"}},"query":"mutation updateDataCenterEventVariable($id: HashId!, $eventVariable: VariableInput!) { updateDataCenterEventVariable(id: $id, eventVariable: $eventVariable) { name __typename }}"}""" % ( var['id'], name, key, var['description'], var['valueType']) elif name == key: body = """{"operationName":"updateDataCenterEventVariable","variables":{"id":"%s","eventVariable":{"name":"%s","key":"%s", "description":"%s","valueType":"%s"}},"query":"mutation updateDataCenterEventVariable($id: HashId!, $eventVariable: VariableInput!) { updateDataCenterEventVariable(id: $id, eventVariable: $eventVariable) { name __typename }}"}""" % ( var['id'], var['name'], key, desc, var['valueType']) else: body = """{"operationName":"updateDataCenterEventVariable","variables":{"id":"%s","eventVariable":{"name":"%s","key":"%s", "description":"%s","valueType":"%s"}},"query":"mutation updateDataCenterEventVariable($id: HashId!, $eventVariable: VariableInput!) { updateDataCenterEventVariable(id: $id, eventVariable: $eventVariable) { name __typename }}"}""" % ( var['id'], name, key, desc, var['valueType']) content = send_graphql_post(token, body.encode('utf-8')) print("事件属性存在且更新成功") else: body = """{"operationName":"createDataCenterEventVariable","variables":{"eventVariable":{"name":"%s","key":"%s", "valueType":"%s","description":"%s"}},"query":"mutation createDataCenterEventVariable($eventVariable: VariableInput!) {createDataCenterEventVariable(eventVariable: $eventVariable) {name __typename}}"}""" % ( name, key, valuetype, desc) content = send_graphql_post(token, body.encode('utf-8')) return content # 创建用户属性 def create_user_variables(token, key, valuetype, name, desc): """ 创建用户属性 """ name = check_name(name, key) check_user_valuetype(valuetype) flag, var = check_user_var_exsit(token, key) if flag: if desc == '': body = """{"operationName":"updateDataCenterUserVariable","variables":{"id":"%s","userVariable":{"name":"%s","key":"%s","description":"%s", "valueType":"%s"}},"query":"mutation updateDataCenterUserVariable($id: HashId!, $userVariable: VariableInput!) { updateDataCenterUserVariable(id: $id, userVariable: $userVariable) { name __typename }}"}""" % ( var['id'], name, key, var['description'], var['valueType']) elif name == key: body = """{"operationName":"updateDataCenterUserVariable","variables":{"id":"%s","userVariable":{"name":"%s","key":"%s","description":"%s", "valueType":"%s"}},"query":"mutation updateDataCenterUserVariable($id: HashId!, $userVariable: VariableInput!) { updateDataCenterUserVariable(id: $id, userVariable: $userVariable) { name __typename }}"}""" % ( var['id'], var['name'], key, desc, var['valueType']) else: body = """{"operationName":"updateDataCenterUserVariable","variables":{"id":"%s","userVariable":{"name":"%s","key":"%s","description":"%s", "valueType":"%s"}},"query":"mutation updateDataCenterUserVariable($id: HashId!, $userVariable: VariableInput!) { updateDataCenterUserVariable(id: $id, userVariable: $userVariable) { name __typename }}"}""" % ( var['id'], name, key, desc, var['valueType']) content = send_graphql_post(token, body.encode('utf-8')) print("用户属性存在且更新成功") else: body = """{"operationName":"createDataCenterUserVariable","variables":{"userVariable":{"name":"%s","key":"%s","valueType":"%s","description":"%s"}}, "query":"mutation createDataCenterUserVariable($userVariable: VariableInput!) {createDataCenterUserVariable(userVariable: $userVariable) { id name __typename }}"}""" % (name, key, valuetype, desc) content = send_graphql_post(token, body.encode('utf-8')) return content def bind_event_variables(token, key, name, attr): """ 绑定事件与事件属性 """ id_list, key_list = check_attr(token, attr) flag, var, attr_id_list = check_event_attr_exsit(token, key) name = check_name(name, key) attr_str = '' size = len(id_list) for i in range(size): attr_str += '{\"id\":\"' + str(id_list[i]) + '\"},' str_sub = attr_str[:len(attr_str) - 1] if flag and len(attr_id_list) > 0: body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s","valueType":"counter", "attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], var['name'], key, var['description'], str_sub) elif flag and len(attr_id_list) == 0: body = """{"operationName":"updateDataCenterCustomEvent","variables":{"id":"%s","customEvent":{"name":"%s","key":"%s","description":"%s","valueType":"counter", "attributes":[%s]}},"query":"mutation updateDataCenterCustomEvent($id: HashId!, $customEvent: CustomEventInput!) { updateDataCenterCustomEvent(id: $id, customEvent: $customEvent) { name __typename }}"}""" % ( var['id'], var['name'], key, var['description'], str_sub) else: body = """{"operationName":"createDataCenterCustomEvent","variables":{"customEvent":{"name":"%s","key":"%s","valueType":"counter", "attributes":[%s]}},"query":"mutation createDataCenterCustomEvent($customEvent: CustomEventInput!) {createDataCenterCustomEvent(customEvent: $customEvent) {name __typename }}"}""" % ( name, key, str_sub) content = send_graphql_post(token, body.encode('utf-8')) return content, key_list def export_meta(token, file): """ 导出元数据 """ meta_data = getMetaData(token) meta_data['events'] = meta_data.pop('dataCenterCustomEvents') meta_data['event_variables'] = meta_data.pop('dataCenterEventVariables') meta_data['user_variables'] = meta_data.pop('dataCenterUserVariables') user_attr_list = list() for i in meta_data['user_variables']: if not str(i['key']).startswith("$"): user_attr_list.append(i) meta_data['user_variables'] = user_attr_list meta_data = str(meta_data).replace('\'', '\"')[:-1] event = getBindEvent(token) event['bind_event_variables'] = event.pop('dataCenterCustomEvents') event = str(event).replace('\'', '\"')[1:] json_str = meta_data + ',' + event with open(file, 'w+', encoding='utf-8') as f: f.write(json_str.encode('utf-8').decode('utf-8')) # json.dump(json.loads(json_str), f) print("导出元数据成功~") def import_meta(token, file): """ 导入元数据 """ with open(file, 'r', encoding='utf-8') as f: json_loads = json.loads(f.read()) if len(json_loads['events']) != 0: for event in json_loads['events']: check_key(event['key']) create_event(token, event['key'], event['name'], event['description']) if len(json_loads['event_variables']) != 0: for var in json_loads['event_variables']: check_key(var['key']) create_event_variables(token, var['key'], var['valueType'], var['name'], var['description']) if len(json_loads['user_variables']) != 0: for var in json_loads['user_variables']: check_key(var['key']) create_user_variables(token, var['key'], var['valueType'], var['name'], var['description']) if len(json_loads['bind_event_variables']) != 0: for var in json_loads['bind_event_variables']: attr_str = '' for attr in var['attributes']: attr_str += str(attr['key']) + ',' sub_attr = attr_str[:-1] if len(sub_attr) > 2: bind_event_variables(token, var['key'], '', sub_attr) print("导入元数据成功~") growingio-tools-python-master/importer/meta_importer.py000066400000000000000000000055201412134335600241160ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import argparse import os import pathlib import sys import logging sys.path.append(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.meta.meta_create import * def parse_args(): parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('-m', help='必填参数. 创建事件-create_event,创建事件属性-create_event_variables,创建用户属性-create_user_variables,绑定事件与事件属性-bind_event_variables,导入元数据-export_meta，导出元数据-import_meta', required=True, metavar="") parser.add_argument('-k', '--key', help='必填参数. 需要创建事件名', default="", metavar="") parser.add_argument('-t', '--type', help='创建事件/用户属性必填参数. 数据类型-(string,int,double)', default="", metavar="") parser.add_argument('-a', '--attr', help='绑定事件与属性必填参数，多个属性名使用英文逗号分隔', metavar="") parser.add_argument('-n', '--name', help='可选参数. 事件显示名', default="", metavar="") parser.add_argument('-d', '--desc', help='可选参数. 事件描述', default="", metavar="") parser.add_argument('-f', '--file', help='导入/导出元数据必填参数. 文件名', metavar="") args = parser.parse_args() return args.__dict__ def main(): args = parse_args() # print(args) m = args.get("m") token = get_token() if 'create_event'.__eq__(m): key = args.get('key') check_key(key) create_info = create_event(token, key, args.get('name'), args.get('desc')) print("创建事件成功", create_info) elif 'create_event_variables'.__eq__(m): key = args.get('key') check_key(key) create_info = create_event_variables(token, key, args.get('type'), args.get('name'), args.get('desc')) print("创建事件属性成功", create_info) elif 'create_user_variables'.__eq__(m): key = args.get('key') check_key(key) create_info = create_user_variables(token, key, args.get('type'), args.get('name'), args.get('desc')) print("创建用户属性成功", create_info) elif 'bind_event_variables'.__eq__(m): key = args.get('key') check_key(key) bind_info, key_list = bind_event_variables(token, key, args.get('name'), args.get('attr')) print("成功绑定事件属性:{},info:{}".format(key_list, bind_info)) elif 'export_meta'.__eq__(m): export_meta(token, args.get('file')) elif 'import_meta'.__eq__(m): import_meta(token, args.get('file')) else: logging.error("请确认填写项目名！") exit(-1) if __name__ == '__main__': main() growingio-tools-python-master/importer/requirements.txt000066400000000000000000000002771412134335600241650ustar00rootroot00000000000000numpy==1.19.1 requests==2.24.0 SQLAlchemy==1.3.19 pandas==1.1.0 hashids==1.3.1 PyYAML==5.3.1 psycopg2-binary==2.8.5 pymysql==1.0.2 pyhive==0.6.3 thrift==0.13.0 sasl==0.2.1 thrift-sasl==0.4.2 growingio-tools-python-master/importer/saas_export.py000066400000000000000000000022321412134335600235740ustar00rootroot00000000000000#!/bin/env python3 # -*- coding: UTF-8 -*- # Copyright (c) 2020 growingio.com, Inc. All Rights Reserved import argparse import os import pathlib import sys import logging sys.path.append(str(pathlib.Path(os.path.abspath(__file__)).parent.parent)) from importer.saas_export.saas_meta import getSaasMeta def parse_args(): parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('-m', help='必填参数. 创建事件-create_event,创建事件属性-create_event_variables,创建用户属性-create_user_variables,绑定事件与事件属性-bind_event_variables,导入元数据-export_meta，导出元数据-import_meta', required=True, metavar="") parser.add_argument('-f', '--file', help='导入/导出元数据必填参数. 文件名', metavar="") args = parser.parse_args() return args.__dict__ def main(): args = parse_args() m = args.get("m") if 'export_saas_meta'.__eq__(m): getSaasMeta(args.get("file")) else: logging.error("请确认填写项目名！") exit(-1) if __name__ == '__main__': main() growingio-tools-python-master/importer/saas_export/000077500000000000000000000000001412134335600232235ustar00rootroot00000000000000growingio-tools-python-master/importer/saas_export/__init__.py000066400000000000000000000000001412134335600253220ustar00rootroot00000000000000growingio-tools-python-master/importer/saas_export/saas_meta.py000066400000000000000000000063451412134335600255420ustar00rootroot00000000000000""" SaaS元数据 """ import json from importer.common.config_util import SaasConfig from importer.common.http_util import send_rest_get def getSaasMeta(file): """ 导出SaaS元数据 """ events = 'v1/api/projects/' + SaasConfig.uid + '/dim/events' event_vars = 'v1/api/projects/' + SaasConfig.uid + '/vars/events' user_vars = 'v1/api/projects/' + SaasConfig.uid + '/vars/peoples' anonymous_var = 'v1/api/projects/' + SaasConfig.uid + '/vars/visitors' parmas = {'project_uid': SaasConfig.uid} events_context = send_rest_get(events, parmas) event_var_context = send_rest_get(event_vars, parmas) usr_var_context = send_rest_get(user_vars, parmas) anonymous_var_context = send_rest_get(anonymous_var, parmas) # 事件 event_list = [] result = {} for data in events_context: event_res = {} event_res['key'] = data['key'].strip() event_res['name'] = data['name'].strip() event_res['description'] = data['description'].strip() event_list.append(event_res) result['events'] = event_list # 事件属性 evar_list = [] for evar in event_var_context: evar_res = {} evar_res['key'] = evar['key'].strip() evar_res['name'] = evar['name'].strip() evar_res['valueType'] = str(evar['type']).lower().strip() evar_res['description'] = '' evar_list.append(evar_res) result['event_variables'] = evar_list # 用户属性 uvar_list = [] uvar_set = set() for uvar in usr_var_context: uvar_res = {} uvar_res['key'] = uvar['key'].strip() uvar_res['name'] = uvar['name'].strip() uvar_res['valueType'] = 'string' uvar_res['description'] = uvar['description'].strip() uvar_list.append(uvar_res) uvar_set.add(uvar['key'].strip()) uvar_size = len(uvar_set) # 访问用户属性 avar_list = [] avar_set = set() for avar in anonymous_var_context: avar_res = {} avar_res['key'] = avar['key'].strip() avar_res['name'] = avar['name'].strip() avar_res['valueType'] = 'string' avar_res['description'] = avar['description'].strip() avar_list.append(avar_res) avar_set.add(avar['key'].strip()) avar_size = len(avar_set) var_set = set() for var in avar_set: if var not in uvar_set: var_set.add(var) if uvar_size + avar_size == len(uvar_set | var_set): result['user_variables'] = uvar_list + avar_list else: for i in avar_list: if i['key'] in var_set: uvar_list.append(i) result['user_variables'] = uvar_list attr_list = [] for data in events_context: event_res = {} list = [] for attr in data['attrs']: item = {} item['key'] = attr['key'].strip() list.append(item) event_res['key'] = data['key'].strip() event_res['attributes'] = list attr_list.append(event_res) result['bind_event_variables'] = attr_list json_str = str(result).replace('\'', '\"') with open(file, "w+", encoding='utf-8') as f: f.write(json_str.encode('utf-8').decode('utf-8')) print("事件，事件属性，用户属性导出成功~") growingio-tools-python-master/importer/setup.py000066400000000000000000000003501412134335600224030ustar00rootroot00000000000000from setuptools import setup, find_packages install_requires = [ ] setup( name='importer', version='1.0', author='GrowingIO', packages=find_packages(include=["importer*"]), install_requires=install_requires )